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I  would  like  to  welcome  Dr.  Amy  Henninger  as  the  newest  member  of  the  MSI  AC  Journal  editorial  board.  Her  editorial 
On  Uncertainty...  is  a  perfect  entre  to  the  theme  of  this  issue  of  the  MSIAC  Journal.  -  Dane  Mullenix,  MSI  AC  Director 


On  Uncertainty... 

As  we  proceed  down  the  path  of  the  "Long  War"  we  are 
facing  a  fundamentally  different  operational  environment 
than  we  faced  during  the  "Cold  War".  Where  we  used  to 
have  a  defined  enemy  with  known  equipment,  tactics, 
and  behaviors  operating  in  a  military-centric  environment, 
we  now  have  emergent  and  adaptive  threats,  and  civil, 
cultural  and  infrastructure  elements  operating  in  a  media¬ 
centric  environment.  These  elements  are  not  well  defined, 
or  even  understood.  As  a  result,  there  is  no  single  best 
method  to  encode  and  describe  the  behaviors  and  the 
results  of  stimuli.  There  are,  instead  and  in  some  cases,  a 
plethora  of  models  that  each  describes  some  element  of 
the  whole.  And,  it  is  generally  accepted  and  understood 
that  these  models  carry  with  them  many  assumptions  and 
uncertainties. 

Uncertainty  is  something  we  have  spent  years  trying  to 
quantify,  minimize  and  avoid  in  our  tools  and  studies.  But,  in 
this"Long  War"  the  existence  of  uncertainty  in  our  modeling 
efforts  is  more  apparent  than  ever,  and  as  we  move  towards 
representing  data-sparse  phenomenan  or  phenomenan 
lacking  well  defined  theoretical  underpinnings,  the  defense 
community  is  forced  to  recognize  that  as  we  design  our 
tools  and  studies  "uncertainty  trumps  utility".  That  is,  as 
uncertainty  about  some  domain  increases,  the  utility  of 
a  traditional  consolidative  representation  of  that  domain 
decreases. 

In  the  first  article  by  Mr.  Chris  DiPetto,  we  see  that 
traditional  methodologies  for  developing  and  testing 
systems  will  fail  to  fully  describe  real  joint  capabilities. 
Advancements  in  evaluation  methodologies  are  required  to 
demonstrate  that  future  systems  will  integrate  seamlessly  in 
complex  battlespace  environments.  Thus,  we  must"test,  as 
well  as  train,  like  we  fight".  In  the  second  paper  by  Dr.  Paul 
Roman  and  Mr.  Doug  Brown,  we  witness  the  application  of 
a  contemporary  technology  to  an  age  old  problem:  How 
to  turn  a  team  of  experts  into  an  expert  team.  And,  in  the 
third  paper  by  Mr.  John  Lawson,  we  consider  a  Marine 
analyst's  perspective  on  the  National  Research  Council's 
latest  review  of  human  behavior  modeling  research. 

Clearly,  we  are  on  the  verge  of  a  paradigm  shift  that  moves 
our  community  from  traditional  consolidative  modeling 


methods  into  next-generation  methods  for  representation 
and  evaluation.  This  is  the  thread  that  runs  through  this 
issue  of  the  MSIAC  Journal:  "It  came  from  the  unknown  - 
the  unpredictable  in  M&S"! 

The  good  news  is  we  are  not  alone!  Other  communities 
have  recognized  this  challenge  and  are  developing 
approaches  to  reduce  uncertainty.  The  meteorological 
modeling  community,  faced  with  potentially  dire 
consequences  for  inaccurate  forecasts  has  developed 
improved  methods  for  dealing  with  uncertainty.  Similarly, 
the  machine  learning  community,  faced  with  the  challenge 
of  learning  very  large  domain  spaces,  has  also  developed 
effective  means  for  reducing  uncertainty.  We  must  do  the 
same.  I  encourage  you  to  explore  the  power  of  computing 
and  computational  modeling  as  a  means  of  reducing 
uncertainty,  like  these  communities  have  done.  Only  by 
accepting  (instead  of  ignoring)  the  uncertainty  in  the 
operational  world  and  by  implementing  methodologies 
that  embrace  the  diversity  of  different  approaches,  will 
we  be  positioned  to  better  serve  our  Warfighters  and  the 
people  who  support  them. 


Amy  E.  Henninger,  Ph.D., 
Associate  Director,  M&SCO 
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Simulation  pervades  our  daily  lives  as  never  before.  Due 
to  the  continuous  evolution  of  computational  capabilities, 
the  powers  of  multi-dimensional,  time-driven  simulation 
environments  are  exploding.  I  propose  that  by  taking 
the  following  three  steps,  we  can  capitalize  on  all  of  this 
"technology"  to  fulfill  the  requirements  and  goals  of  our 
simulation-related  endeavors. 

Step  One:  As  a  developer,  first  ask  what  the  user  wants. 
The  user  is  that  unique  person  who  should  know  best 
what  is  wanted  from  a  simulation.  However,  the  user 
does  not  always  appreciate  or  understand  the  full 
capabilities  of  computer  systems  or  tools  at  hand. 

Step  Two:  Find  a  way  to  help  the  user  describe  what  is 
needed  and  then  to  see  how  the  software  or  simulation 
development  system  can  help  realize  that  need.  This 
requires  a  robust  conceptual  modeling  toolkit  that 
creates  visualizations  of  processes,  entity  behaviors 
and  system  interactions.  Conceptual  models  form  the 
foundation  of  agreement  between  the  user  and  the 
developer.  This  step  should  employ  a  unified  modeling 
language  or  graphical  interface  with  a  wealth  of 
diagrams  to  depict  the  user's  goals  and  objectives. 
Both  parties  will  then  adjust  and  build  a  final  view  of 
the  simulation's  concept  on  paper. 

StepThree:  Oncethedeveloperanduserarecomfortable 
with  the  way  the  system  is  shaping  up,  then  it's  time 
to  choose  and  use  a  programming  language  or 
environment.  The  programmer  can  start  from  scratch 
or  reuse  components  from  existing  simulations  or 
runtime  environments.  But,  I  caution — step  one  and 
step  two  must  be  completed  first. 

Now  that  you  have  the  basis  for  building  a  simulation  in 
a  high-tech  environment,  you're  ready  to  read  the  rest  of 
this  issue  of  the  MSIAC  Journal.  This  issue  provides  insights 
into  three  diverse,  yet  interrelated,  areas  in  modeling 
and  simulation.  Following  the  spectrum  from  test  and 
evaluation  of  our  future  joint  environments,  to  new  gaming 
implementations,  to  soft  modeling  of  abstract  modern 
concepts,  the  papers  cover  a  wide  expanse  of  important 
M&S  subjects. 

First  on  the  list  is  Mr.  Chris  DiPetto's  explanation  and 
review  of  how  a  Live,  Virtual  and  Constructive  Distributed 
Environment  enabled  joint  mission  event  testing  and 
evaluation  for  command  and  control  systems,  including 
the  Army's  Future  Combat  System.  Dr.  Paul  Roman  and 
Mr.  Doug  Brown  from  Canada  articulate  serious  gaming 


concerns  and  issues  that  have  intrigued  many  of  us  in  the 
simulation  industry  for  years.  Finally,  the  idea  of  Irregular 
Warfare  M&S  as  related  to  behavior  modeling  is  part  of  John 
Lawson's  premise  that  IW  modeling  is  often  harder  than 
thought,  but  does  not  have  to  be  too  complex  to  succeed. 
In  these  globally  turbulent  times  with  the  Quadrennial 
Defense  Review  upon  us,  this  is  even  more  relevant. 

In  summary,  the  MSIAC  team  is  confident  that  as  you 
immerse  yourself  in  this  set  of  thought-provoking  articles, 
you  will  learn  how  simulation-based  environments  enable 
and  enhance  mission  accomplishment. 


George  F.  Stone,  Ph.D 

Senior  Scientist  for  Modeling  and  Simulation 
Alion  Science  and  Technology 
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Test  and  Evaluation 

Paving  the  Way  for  Testing  in  a  Joint  Environment 
The  Capability  Test  Methodology 
Chris  DiPetto 

The  very  nature  of  modern  warfare  necessitates  major 
changes  to  the  way  the  Department  of  Defense  tests  and 
acquires  systems  and  capabilities.  Since  most  systems 
today  are  deployed  in  joint  environments,  the  testing 
of  a  system  by  a  single  Service  may  not  be  adequate 
to  demonstrate  that  the  system  meets  the  warfighter's 
needs.  Future  systems,  which  are  expected  to  operate  in 
a  joint  environment,  should  be  tested  in  a  realistic  joint 
environment  throughout  the  acquisition  life  cycle,  starting 
with  early  experimentation  and  concept  development 
through  the  developmental  and  operational  test. The  result 
is  an  optimally  integrated  system. 

DoD  needs  to  ensure  that  it  is  testing  systems,  systems 
of  systems  (SoS),  and  capabilities  consistent  with  their 
intended  use.  In  other  words,  we  test,  as  well  as  train,  like  we 
fight.  The  warfighter  should  be  confident  the  systems  work 
as  advertised,  and  the  tester  must  be  challenged  to  deliver 
the  future  joint  capabilities  needed  by  the  warfighter. 

A  New  Roadmap 

Changes  in  testing  and  acquisition  processes  are  under 
way  to  make  this  happen.  DoD  instructions  acknowledge 
the  need  to  test  joint  capabilities  in  the  expected  joint 
operational  environment. The  Joint  Capabilities  Integration 
and  Development  System  (JCIDS)  is  applying  capabilities- 
based  approaches  to  transform  the  way  the  DoD  defines 
requirements  for  new  systems  and  capabilities  by  moving 
materiel  developers  and  testers  away  from  the  Service¬ 
centric  system  requirements  of  the  past  and  toward 
the  necessary  joint-centric  capability  development 
for  future  systems.  In  November  2004,  in  response  to 
strategic  planning  guidance  direction  to  provide  new 
testing  capabilities  and  institutionalize  the  evaluation 
of  joint  system  effectiveness  as  part  of  new  capabilities- 
based  processes,  then-Deputy  Secretary  of  Defense  Paul 
Wolfowitz  approved  the  Testing  in  a  Joint  Environment 
Roadmap,  developed  by  the  DoD  director  of  operational 


test  and  evaluation.  The  roadmap  calls  for  actions  that 
establish  a  framework  for  the  life  cycle  evaluation  of 
systems  and  SoS  in  a  joint  operational  environment 
beginning  with  the  JCIDS  process.  Implementation  of  the 
roadmap  focuses  on  three  elements — policy,  infrastructure 
networks  and  middleware,  and  methods  and  processes — 
while  recognizing  the  important  role  of  the  Services  and 
agencies  in  the  execution.  The  purpose  of  the  roadmap  is 
to  coordinate  and  synchronize  the  sometimes  disparate 
Service  and  agency  testing  efforts  by  capitalizing  on 
existing  Service  test  assets,  forming  an  approach  to  joint 
testing  that  will  ensure  systems  and  capabilities  function 
as  intended  when  integrated  into  the  joint  mission 
environment. 

As  individual  platforms  become  part  of  a  complex, 
networked  SoS  that  must  work  effectively  in  a  joint 
battlespace,  effective  test  and  evaluation  is  becoming 
more  difficult.  In  the  future,  programs  expected  to  operate 
in  a  joint  environment  should  demonstrate  their  joint 
capability  early  and  throughout  their  developmental 
cycles,  regardless  of  the  program's  acquisition  category. 
The  objective  of  the  roadmap  is  to  address  these  challenges 
and  define  changes  that  will  position  test  capabilities  to 
fully  support  adequate  test  and  evaluation  of  warfighting 
capabilities  developed  under  DoD's  capabilities-based 
processes  in  the  appropriate  joint  mission  environment. 

The  joint  mission  environment  provides  the  operational 
context  in  which  the  capability  being  developed  must 
perform.  Important  aspects  of  this  operational  context 
include  joint  mission,  task,  threat  condition,  environmental 
condition,  and  system  or  SoS  descriptions  of  capabilities 
supporting  the  joint  mission. The  joint  mission  environment 
is  realized  when  all  relevant  aspects  of  the  joint  operational 
context  are  adequately  represented  in  an  environment 
ready  for  a  test  that  may  be  live,  virtual,  and/  or  constructive 
and  distributed  in  nature. 

The  Test  Resource  Management  Center,  established  as  a 
field  activity  reporting  to  the  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics,  is 
chartered  to  be  the  steward  of  DoD  test  and  evaluation 
infrastructure.  In  that  capacity  TRMC  is  responsible  for 
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the  Joint  Mission  Environment  Test  Capability  Program 
to  help  establish  a  DoD-wide  live,  virtual,  constructive 
distributed  environment  (LVC-DE).  To  date,  JMETC  has 
laid  the  groundwork  for  an  enterprise-level  solution  to 
testing  infrastructure  and  has  enabled  that  infrastructure 
to  be  assembled  more  quickly  than  in  the  past.  Even  with 
an  effective  infrastructure  in  place,  there  is  also  a  need  for 
policy  changes  and  new  methods  and  processes  to  make 
valid  testing  in  a  joint  mission  environment  a  reality. 

The  Methods  and  Processes 

Traditional  methodologies  for  developing  and  testing 
military  systemsexpectedtooperateinajointenvironment — 
particularly  verifying  specification  compliance  for  individual 
systems  and  testing  within  a  single  Service  environment — 
will  fail  to  fully  describe  real  joint  capabilities.  As  DoD 
moves  away  from  traditional  single-system  approaches  to 
new  joint  capability-based  approaches,  the  department 
must  demonstrate  that  its  future  weapons  integrate 
seamlessly  into  SoS  and  capabilities  in  complex  battlespace 
environments  to  produce  coordinated  and  focused  effects. 
In  addition,  doctrine,  organization,  training,  leadership, 
personnel,  and  facility  aspects  of  these  new  capabilities 
along  with  materiel  needs  must  be  addressed  early  in  the 
development  process.  DoD's  long-term  strategy  calls  for 
evaluations  of  joint  systems  effectiveness  throughout  all 
phases  of  a  capability's  development  and  deployment.  As  I 
mentioned  before,  we  want  to  be  able  to  test  as  we  fight. 

The  challenge  of  testing  in  a  joint  mission  environment 
begins  early  in  the  system  acquisition  life  cycle;  and  it  is 
daunting  considering  the  number  of  systems,  network 
combinations  and  interactions,  environmental  conditions, 
and  non-materiel  aspects  that  must  be  addressed  for  a 
realistic  test.  How  much  testing  and  data  are  sufficient?  Must 
every  possible  combination  of  environmental  conditions, 
modes  of  operation,  systems,  and  entities  within  the  joint 
mission  environment  be  exercised?  Replicating  a  realistic 
joint  environment  will  be  very  challenging  because  the 
ability  to  assemble  all  required  assets  at  a  single  test  location 
will  be  nearly  impossible  because  of  scheduling  constraints 
and  resource  availability.  How  much  of  this  environment 
needs  to  be  available  for  developmental  testing  and  how 
much  is  required  for  operational  testing?  How  can  a  realistic 


joint  environment  be  constructed  to  enable  it  to  meet 
both  developmental  and  operational  test  objectives?  What 
kinds  of  tests  can  be  done  during  developmental  testing 
to  reduce  the  risk  of  uncovering  new  system  deficiencies 
during  operational  testing? 

To  address  some  of  these  challenges  as  part  of  the 
larger  roadmap  effort,  the  director  of  operational  test 
and  evaluation  chartered  the  Joint  Test  and  Evaluation 
Methodology  project  in  2006.  Specifically,  JTEM  was 
directed  to  develop,  test,  and  evaluate  methods  and 
processes  for  defining  and  using  an  LVC-DE  joint  test 
environment  to  evaluate  system  performance  and  joint 
mission  effectiveness. 

JTEM  has  developed  the  capability  test  methodology, 
which  is  a  collection  of  recommended  best  practices  for 
designing  a  test  of  a  system  or  SoS  in  a  complex  joint 
environment.  The  CTM  provides  a  rational  process  that 
guides  the  program  manager  and  test  manager  through 
the  test  planning  process  to  tailor  and  optimize  a  test  to 
demonstrate  system  performance  within  a  joint  context 
as  well  as  system  contribution  to  joint  capabilities.  It  is  a 
foundation  for  a  series  of  guides,  handbooks,  and  training 
courses  that  will  ultimately  bedelivered  to  test  organizations 
and  acquisition  PMs.The  CTM  is  intended  to: 

•  Address  testing  of  systems,  SoS,  and  capabilities,  be 
they  Service  or  joint 

•  Augment  existing  DoD  and  Service  test  processes  1 1 
Defense  AT&L:  September-October  2008 

•  Align  test  and  evaluation  aspects  and  information 
across  multiple  DoD  processes,  namely  Analytic 
Agenda,  JCIDS,  DoD  Architecture  Framework,  and  the 
Defense  Acquisition  System 

•  Provide  recommended  best  practices  for  a  consistent 
approach  to  describing,  building,  and  using  an 
appropriate  representation  of  a  particular  joint 
mission  environment  across  the  acquisition  life  cycle 

•  Reflect  current  acquisition  policies  and  instructions, 
and  eventually  be  incorporated  into  Defense 
Acquisition  University  PM  and  test  and  evaluation 
courses. 

The  CTM  is  designed  to  augment,  not  replace,  existing 
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test  methods  and  processes,  taking  into  account  the  unique 
aspects  of  testing  joint,  networked  systems  in  an  LVC-DE.  As 
such,  the  CTM  closely  parallels  existing  test  processes  used 
within  DoD.The  CTM  consists  of  six  steps  and  14  processes, 
which  are  briefly  described  in  the  following  paragraphs. 

The  CTM  Steps 

Step  0  defines  the  test  evaluation  strategy.  The  key 
process  in  this  step  is  describing  the  Joint  Operational 
Context  for  Test  used  to  define  the  specific  elements  that 
make  up  the  LVC-DE.  The  Joint  Operational  Context  for  Test 
includes  a  detailed  description  of  the  system  under  test, 
supporting  systems,  the  expected  operating  environment, 
threat  forces,  and  key  system  interactions  and  information 
exchanges  required  to  complete  a  particulartaskor  mission. 
In  step  1 ,  the  PM  creates  a  program  introduction  document, 
which  outlines  the  details  of  a  particular  test  or  set  of  tests, 
communicating  requirements  to  a  test  range.  The  test 
range  then  uses  that  document  to  produce  a  statement  of 
capability,  which  is  the  starting  point  for  determining  what 
resources  will  be  used  to  conduct  the  test  and  what  data 
will  be  collected.  Step  2  produces  distributed  test  plans 
that  are  compilations  of  current  individual  test  plans,  with 
the  addition  of  distributed  and  joint  elements.  During  this 
test  planning  phase,  early  test  concepts  are  developed  into 
more  detailed  test  plans.  Test  planning  processes  include 
test  trial/vignette  selection,  refining  the  live,  virtual, 
constructive  distributed  test  environment  required  and 
synthesizing  these  activities  into  a  test  plan. 

During  CTM  steps  3, 4,  and  5,  joint  mission  environments 
are  assembled  and  used  to  support  multiple  test  plans. 
Step  3  is  concerned  with  technical  systems  engineering 
activities  for  automatic  distributed  LVC-DE  implementation. 
These  processes  include  the  design  of  distributed 
configurations,  the  assembly  of  distributed  components, 
and  the  integration  of  components  into  a  distributed 
test  range  that  meets  customer  requirements.  In  CTM 
step  4,  distributed  tests  are  conducted  according  to  local 
procedures  and  data  are  collected. This  phase  produces  test 
data  for  customers  and  reusable  information  for  future  joint 
mission  environments.  Though  joint  mission  environments 
are  assembled  to  support  multiple  customers,  tests  are 
not  required  to  run  concurrently.  Sometimes  individual 


customers  may  separately  schedule  only  those  parts  of  the 
joint  mission  environment  they  need  to  meet  their  own 
objectives  for  testing  in  a  joint  environment.  Other  times, 
multiple  customers  may  share  a  joint  mission  environment 
at  the  same  time  for  convenience  or  other  reasons.  In  step  5, 
data  sets  are  processed,  analyzed,  and  evaluated  (including 
evaluations  of  joint  mission  effectiveness  and  contributions 
of  individual  systems  to  joint  missions). 

Supporting  Measures  Framework 

In  addition  to  the  CTM,  JTEM  has  developed  a  supporting 
measures frameworkthat  establishes  appropriate  measures 
to  support  the  evaluation  of  a  system  or  SoS  within  a 
capabilities  context.  This  framework  is  based  on  the  JCIDS 
definition  of  a  capability:  "The  ability  to  achieve  a  desired 
effect  under  specified  standards  and  conditions  through  a 
combination  of  ways  and  means  to  perform  a  set  of  tasks." 
Measures  of  effectiveness  are  established  at  the  mission 
level  and  based  on  combatant  commander-desired  joint 
mission  effects.  The  joint  mission  effects  are  documented 
through  a  compilation  of  products  that  make  up  the  Analytic 
Agenda,  which  is  a  DoD-wide  framework  for  analyzing 
force  structure  requirements  and  other  analytic  studies. 
The  products  used  to  document  desired  effects  include  the 
defense  planning  scenarios,  which  are  a  series  of  scenarios 
that  describe  the  range  of  military  operations  for  which 
combatant  commanders  must  be  prepared,  along  with  the 
operating  forces  and  threats  described  in  the  multi-Service 
force  deployment  database  and  the  current  year  and 
future  year  analytical  baselines.  The  desired  effects  must  be 
achieved  under  specified  standards  and  conditions  using 
systems;  SoS;  and  the  supporting  doctrine,  organization, 
training,  leadership,  personnel,  and  facility  aspects,  which 
make  up  the  combinations  of  ways  and  means. 

The  systems  and  SoS  have  various  performance  attributes 
associated  with  them  (e.g.,  launch  range  of  an  aircraft  or 
time  to  disseminate  information  to  the  battlefield  from 
a  higher  echelon  headquarters),  and  they  are  ultimately 
are  used  to  perform  a  set  of  joint  tasks  that  achieve  the 
joint  mission  desired  effects.  In  the  measures  framework, 
measures  of  performance  are  used  to  describe  the  overall 
performance  desired  for  each  particular  task.The  joint  tasks 
are  described  through  the  Universal  Joint  Task  List  and  the 
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Joint  Mission  Essential  Task  List,  along  with  the  specified 
standards  and  conditions.  The  Universal  Joint  Task  List  and 
the  Joint  Mission  EssentialTaskListalso  have  corresponding 
Service  task  lists  that  support  them.  Although  mission 
measures  of  effectiveness  will  be  difficult  to  capture  directly 
during  tests  in  a  joint  environment,  the  task-level  measures 
of  performance  and  the  system  and  SoS  attributes  can  be 
readily  measured.  Analysis  and  combat  modeling  can  then 
be  used  to  determine  overall  measures  of  effectiveness  for 
the  joint  mission  desired  effects. 

Testing  in  a  Realistic  Environment 

The  CTM  is  addressing  the  ways  and  means  for  designing 
and  executing  tests  of  complex,  networked  SoS  in  a  realistic 
joint  mission  environment  through  its  newly  developed 
and  enhanced  methods  and  processes.  During  their  2007 
test  event,  JTEM  demonstrated  application  of  the  CTM  to  a 
notional  set  of  network-enabled  air-  and  ground-launched 
weapon  systems  while  employed  in  a  joint  mission 
environment  supporting  a  joint  fire  support  task.  This  test 
showed  the  potential  that  can  be  realized  from  testing 
networked  SoS  in  a  realistic  operational  environment. 
However,  it  also  revealed  many  challenges,  which  fall  into 
the  categories  of: 

•  Agreed-to  measures  of  performance  and  effectiveness 
across  multiple  joint  missions 

•  Persistence  of  the  test  environment  used  for  testing 

•  Analysis  and  data  management  techniques  to  deal 
with  the  increasing  complexity  of  planning  tests  and 
evaluating  the  results  of  tests  in  net-centric  systems. 

JTEM  continues  to  address  these  issues.  In  this  year's 
test  event,  the  CTM's  effectiveness  and  suitability  for  use 
in  a  complex  joint  test  environment  was  again  assessed 
using  the  Army's  Combined  Test  Organization  for  Future 
Combat  Systems' Joint  Battlespace  Dynamic  Deconfliction 
event  as  a  joint  capability  test  event.  This  venue  provided 
the  opportunity  to  identify  the  challenges  in  integrating 
the  end-to-end  CTM  into  existing  test  activity,  developing 
and  maturing  the  LVC  distributed  prototype,  investigating 
data-requirements  issues,  analyzing  deficiencies  in  the  joint 
mission  environment  representation  requirements,  and 
assessing  the  LVC  distributed  environment  instantiation 


of  the  joint  mission  environment.  Test  event  results  are 
driving  improvements  to  the  CTM  and  will  provide  an 
opportunity  to  gain  a  better  understanding  of  what  it 
takes  to  fully  realize  a  sufficient  capability  to  test  in  a  joint 
environment  across  DoD.  In  April  2009,  JTEM  will  deliver 
a  version  of  the  CTM,  along  with  guides,  handbooks,  and 
additional  supporting  documents,  which  will  prepare  PM 
and  test  organization  customers  to  effectively  test  as  the 
capabilities-based  approach  to  acquisition  requires. 

Continuous  Learning  Available 

Additional  work  has  been  accomplished  to  facilitate  our 
future  testing  needs.  The  Defense  Acquisition  University 
and  JTEM  have  partnered  to  develop  a  Testing  in  a  Joint 
Environment  continuous  learning  module,  now  available 
on  the  DAU  Web  site  <http://clc.dau.mil/>.  The  module's 
goal  is  to  familiarize  DoD  personnel  with  basic  principles 
and  practices  related  to  testing  in  a  joint  environment.  This 
three-hour  credit  course  will  enable  capability  managers, 
PMs,  requirements  managers,  systems  engineers,  test  and 
evaluation  professionals,  acquisition  professionals,  and 
warfighters  to: 

•  Recognize  the  need  for  testing  in  a  joint  environment 

•  Describe  the  key  DoD-level  concepts  that  support 
testing  in  a  joint  environment 

•  Describe  the  generalized  methodology  for  testing  in  a 
joint  environment 

•  Define  the  structured  approach  for  identifying 
measures  that  support  testing  in  a  joint  environment 

•  Recognize  the  features  of  the  joint  mission 
environment. 

The  future  of  testing  in  a  joint  environment  has  many 
challenges  and  many  exciting  opportunities.  Through  the 
work  of  the  Testing  in  a  Joint  Environment  Roadmap  and 
the  efforts  of  all  in  the  testing  and  acquisition  communities, 
the  challenges  will  be  met  and  the  opportunities  will  be 
exploited. This  collective  effort  will  enable  us  to  test  like  we 
fight  and  deliver  the  future  joint  capabilities  needed  by  the 
warfighter. 
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Few  would  argue  that  the  pedagogical  advantages  and 
impressive  levels  of  resolution  offered  by  the  latest  in 
video  game  technology  make  it  clear  that  serious  games 
(SG)  have  a  role  to  play  in  military  tactical  training.  Trainers 
close  to  the  front  lines  have  started  adopting  and  adapting 
these  tools  to  meet  real  and  urgent  training  requirements. 


As  military  forces  around  the  world  begin  to  adopt 
gaming  technology  as  an  apparently  cost  effective 
and  robust  means  for  military  tactical  training  it  seems 
appropriate  to  consider  how  well  suited  they  are  for  this 
task.  This  paper  uses  an  evidence  based  approach  to 
illustrate  how  American,  British,  Canadian  and  Australian 
forces  are  applying  serious  game  (SG)  technology  to  meet 
a  variety  of  training  needs.  In  particular,  the  paper  uses 
these  specific  examples  to  address  three  questions:  What 
tactical  training  requirements  are  serious  games  best 
suited  to  meeting?  How  effective  and  efficient  are  they  at 
meeting  those  requirements?  What  are  the  technological 
limits  associated  with  their  use? 

In  answering  these  questions,  the  paper  concludes 
that  SGs  are  providing  a  cost-effective  means  to  provide 
experience-based  learning  with  emphasis  on  cognitive 
and  increasingly  affective  training  domains.  War  fighters 
will  not  develop  the  expert  psycho-motor  skills  they  need 
to  effectively  employ  their  weapon  systems  using  game- 
based  training.  However,  once  the  team  of  experts  in 
various  weapon  systems  is  created,  SG  technology  affords 
trainers  the  opportunity  to  turn  them  into  an  expert  team 
capable  of  communicating  well  with  the  cognitive  skills 
they  need  to  effectively  operate  as  teams.  The  examples 
demonstrate  that  this  is  true  for  infantry,  armoured  or 
combined  arms  training  in  open  or  urban  terrain  and  holds 
for  the  very  technologically  demanding  case  of  aviation 
training.  To  take  full  advantage  of  this  capability,  SGs  need 
to  be  included  as  part  of  blended  training  solutions  that 
take  advantage  of  the  strengths  of  the  various  types  of 
training  available  with  the  SGs  providing  an  experience- 
based  learning  alternative  that  has  not  been  practically 
affordable  since  the  end  of  the  Cold  War. 


In  the  Canadian  Forces,  for  example,  several  training 
establishments  are  using  their  own  budgets  to  acquire 
these  surprisingly  affordable  software  programs.  There  is 
no  shortage  of  choice  either  as  the  video  game  industry 
comes  to  appreciate,  what  from  their  perspective  might  be 
perceived  as  a  niche  market,  an  opportunity  to  differentiate 
their  products  to  meet  the  special  needs  of  the  military 
training  market.  Free  trial  licences  and  a  willingness 
to  accept  feedback  and  make  improvements  are  good 
business  practices  for  these  companies  as  they  incorporate 
the  needs  of  military  users  into  products  that,  as  a  result  of 
the  increased  realism,  appeal  to  a  much  broader  audience. 
One  need  look  no  further  than  the  "Serious  Games 
Showcase  and  Challenge"  now  in  its  third  year  at  the  l/ITSEC 
Conference  to  appreciate  the  broad  scope  of  potential 
applications  for  military  training.  Furthermore,  the  2007 
l/ITSEC  Special  Event  -  "DoD  Training  -  Impact  of  Gaming 
Technologies"  was  aimed  at  helping  DoD  determine  what 
specific  training  needs  could  be  filled  with  the  application 
of  game  technology  and  gave  serious  games  practitioners 
an  opportunity  to  discuss  how  the  industry  might  best 
meet  these  needs. 

Urlocker  and  Smith  (2007)  chronicle  the  recent  history 
of  video  game  technology  in  the  US  Army  and  how  it  has 
grown  to  become  a  disruptive  technology  augmenting 
or  replacing  many  of  the  less  flexible  and  more  expensive 
training  technologies  of  the  past.  From  the  failed  initial 
attempt  with  "Marine  Doom"  through  the  marketing 
success  of  "America's  Army"  the  authors  describe  how 
and  why  video  game  technology  has  found  its  way  into 
the  main  stream.  This  phenomenon  is  not  limited  to 
the  military  however,  as  explained  by  Smith  (2007)  who 
describes  the  five  forces  that  are  driving  the  adoption  of 
game  technology  within  multiple  established  industries 
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of  which  the  military  is  only  one.  Borrowing  from  the 
Five  Forces  Model  popularized  by  Michael  Porter  (Porter, 
1979)  Smith  uses  an  adaptation  of  this  model  to  describe 
a  traceable  adoption  pattern  that  can  be  easily  mapped  to 
the  US  Army  experience  with  SGs  (see  Figure  1 ).  It  includes 
the  niche  market  success  (for  recruiting)  achieved  through 
America's  Army  which  provided  the  technology  a  new 
legitimacy  or  "foot  in  the  door"  that  afforded  innovators 
the  opportunity  to  explore  other  options  based  upon 
an  already  legitimized  technology.  The  next  step  in  the 
adoption  pattern,  as  described  by  Smith,  is  for  applications 
to  be  certified  for  use  by  training  authorities.  A  recent 
report  on  Coalition  Simulation  Interoperability  in  support 
of  the  American,  British,  Canadian  and  Australian  (ABCA) 
armies' program  highlighted  that  VBS  2®  and  OneSAF  were 
the  two  simulation  systems  that  all  coalition  armies  were 
using  or  planning  to  use  in  the  near  future  (Roman,  et.  al., 
2007).  This  report  argued  for  common  tools  as  the  best 
means  to  achieve  simulation  interoperability  and  proposed 
a  SG  (VBS  2®)  and  OneSAF  as  the  current  lead  candidates  for 
coalition  simulation  interoperability.  The  recommended 
practices  in  this  report  may  lead  to  the  last  step  in  Smith's 
adoption  pattern  -  mandatory  standards.  However,  this 
may  be  several  years  away  as  the  experimentation  force 
(one  of  Smith's  5  forces)  is  likely  to  continue  to  influence 
the  adoption  pattern  in  the  near  future.  Part  of  that 
experimentation  is  associated  with  determining  the  most 
effective  ways  to  use  game  technology  in  support  of 
military  tactical  training. 

Pringle  (2007)  also  supports  the  use  of  game  technology 
in  military  training.  He  suggests  that  the  extent  to  which 
SGs  are  expected  to  play  an  increasing  role  in  training 
will  depend  on  an  ability  to  blend  technologies  in  such  a 
way  that  the  training  benefit  is  maximized.  This  implies 
a  requirement  to  be  able  to  measure  the  improvements 
achieved  through  the  adoption  of  these  technologies  and 
a  willingness  to  experiment  with  them.  Unfortunately 
there  are  very  few  well  defined  or  accepted  standards  for 
the  specific  measurement  of  the  effectiveness  of  SGs  and 
few  military  organizations  conduct  the  required  studies 
but  rather  seem  to  accept  their  use  on  faith. 

One  might  speculate  as  to  why  this  is  the  case,  but 
there  is  considerable  anecdotal  evidence  to  suggest  that 


commanders  simply  trust  their  intuition  in  terms  of  the 
effectiveness  of  these  tools.  For  example,  in  preparation 
for  an  Afghanistan  deployment,  a  Canadian  Battle  group 
employed  the  SGVBS®from  Bohemia  Interactive.  As  soon  as 
the  commander  saw  the  value  in  the  pilot  implementation 
at  the  home  station,  he  decided  to  integrate  the  tool 
into  his  Battle  Group's  high  readiness  training.  It  was  by 
no  means  seen  as  a  replacement  to  other  training,  but 
rather  an  enhancement  that  allowed  the  squads,  sections 
and  platoons  to  develop  a  cognitive  understanding  of 
the  tactics  techniques  and  procedures  (TTPs)  for  various 
scenarios  before  executing  them  as  part  of  live  training. 
Difficult  to  practice  scenarios  such  as  convoy  training  were 
also  greatly  facilitated  through  the  use  of  the  game  (Cote, 
2007).  The  commander  in  this  case  clearly  used  his  intuition 
to  develop  a  blended  training  solution  emphasizing  the 
game  for  cognitive  and  difficult  to  practice  skills  combined 
with  traditional  live  training  to  meet  his  specific  needs. 

The  examples  above  illustrate  that  SGs  are  being  used 
for  some  very  serious  purposes.  There  is  a  growing  body 
of  evidence  to  support  the  effectiveness  of  these  tools, 
but  there  appears  to  be  a  lack  of  overall  guidance  on  their 
strengths  and  current  limitations. 

In  considering  the  question:"How  Serious  Are  they?"This 
paper  will  attempt  to  address  the  following: 

•  What  tactical  training  requirements  are  serious  games 
best  suited  to  meeting? 

•  How  effective  and  efficient  are  they  at  meeting  those 
requirements? 

•  What  are  the  technological  limits  associated  with  their 
use? 

What  Tactical  Training  Requirements  are  Serious  Games 
Best  Suited  to  Meeting? 

The  current  operational  tempo  within  the  Canadian 
Army  has  forced  training  policy  and  planning  organizations 
to  become  increasingly  outcome  oriented.  While  there 
is  a  great  deal  of  interest  in  all  of  the  specific  tools  that 
are  available,  the  pressures  of  limited  resources  (most 
importantly,  time,  personnel  and  money)  have  created  a 
very  strong  desire  for  a  systems  view  of  both  requirements 
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Figure  1:  Five  Forces  behind  the  Adoption  of  Game  Technologies  by  Diverse  Industries 

(Smith,  2007) 


and  the  means  available  to  achieve  them.  The  training 
needs  framework  (TNF),  was  created  to  provide  this  view 
(Roman  and  Bassarab,  2007).  Figure  2  displays  where  SGs, 
are  being  used  to  support  Canadian  Army  tactical  training 
from  individual  (level  1)  to  company  (level  5) 


In  the  Canadian  Army,  the  process  of  preparing  forces 
for  operational  deployment  is  known  as  the  "Road  to  High 
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Figure  2:  Application  domain  for  SGs  on  theTraining  Needs 
Framework  (Roman  and  Bassarab,  2007) 


Readiness"  (RHR).  In  many  respects,  the  RHR  is  redesigned 
for  each  organization  that  goes  through  the  process.  It 
does,  however,  follow  a  relatively  standard  progression 
from  individual  skills  to  small  team  skills,  combined  arms 
teams  and  eventually  full  battle  group  tasks  in  the  context 
of  a  brigade  level  operation.  It  does  this  by  using  the  list 
of  Battle  Task  Standards  (Canadian  equivalent  of  Mission 
Essential  Task  List)  as  the  guide  for  the  required  capabilities. 
The  culminating  activity  is  a  confirmation  event  conducted 
as  live  training  at  the  Canadian  ManoeuvreTraining  Centre 
(CMTC)  in  Wainwright,  Alberta. 

The  RHR  process  uses  experiential  learning  as  its 
foundation.  Menaker,  Coleman,  Collins  and  Murawski 
(2006)  summarized  the  recent  research  on  experiential 
learning  as  it  relates  to  military  training  and  advocated  a 
structured  learning  cycle  upon  which  to  base  the  process. 
The  authors  stated  that  experiential  learning  events  must: 

•  Engage  the  learner  mentally. 

•  Emulate  real-world  environments.  Real-world  refers  to 
the  physical  environment  and  the  cognitive  tasks. 

•  Allow  the  learner  to  experience  effects  of  the 
decisions. 
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•  Require  the  learner  to  reflect  on  outcomes  of  their 
actions.  Build  on  established  military  practices  of 
debriefs,  lessons  learned,  and  after  action  reviews. 

•  Revisit  experiences  with  increasing  levels  of  complexity 
to  expand  the  learners'  knowledge  and  skills  by 
increasing  the  number  of  events,  pace  and  emotional 
intensity. 

The  TNF  supports  this  list  of  requirements.  However, 
it  goes  further  to  provide  a  specific  structure  against 
which  an  appropriate  learning  cycle  can  be  established. 
For  example,  the  cycle  builds  from  basic  skills,  such  as 
small  arms  proficiency,  to  discrete  vignettes  in  which  the 
skills  are  practiced  within  various  contexts.  One  specific 
example  of  this  would  be  convoy  operations.  It  then  builds 
to  continuous  scenarios  in  which  the  learners  receive 
increasing  levels  of  cognitive  loading  where  they  must 
decide  which  actions  are  appropriate  without  knowing  the 
overall  context  as  would  have  been  the  case  in  the  discrete 
vignette.  Roman  and  Bassarab  (2007)  provide  a  specific 
example  that  illustrates  the  role  of  SGs  as  part  of  an  overall 
training  plan  for  rules  of  engagement  training.  From  this 
example,  the  authors  show  that  SGs  provide  an  excellent  fit 
with  the  requirements  described  above. 

In  addition  to  the  Canadian  Experience,  British  and 
American  researchers  are  arriving  at  similar  conclusions.  In 
a  series  of  studies  funded  by  the  UK  Acquisition  Research 
Organization,  QinetiQ  researchers  demonstrated  that 
training  could  be  enhanced  for  dismounted  infantry 
section  and  fire  team  TTPs.  Anatolik  (2005)  went  further 
to  examine  how  much  synthetic  environment  (SE)  based 
training  was  appropriate  in  terms  of  a  balance  with 
live  training.  Although  this  study  was  limited  to  urban 
environments,  users  and  trainers  agreed  that  the  video 
game  based  Dismounted  Infantry  Virtual  Environment 
(DIVE),  using  the  commercial  game  engine  Half  Life,  was 
good  for  the  following: 

•  Introducing,  teaching  and  rehearsing  new  drills  and 
TTPs. 

•  Showing  the  viewpoint  of  both  sides,  enemy  and  own 
forces. 

•  Representing  the  use  and  effects  of  current  and 
future  systems  that  either  cannot  be  or  are  poorly 


represented  in  conventional  training. 

•  Reviewing  actions  and  events  from  all  perspectives 
both  during  the  event  and  in  post  game  analysis. 

•  After  Action  Review  (AAR).  This  was  reported  as  a  'big 
win'and  develops  a  feeling  of  inclusion  in  the  training 
process  for  all  participants. 

•  Developing  new  teams  and  fostering  teamwork. 

It  was  also  reported  that  the  representation  of  weapons 
and  systems  capabilities  that  cannot  be  represented  in 
conventional  urban  training  was  extremely  valuable. 
These  included  grenades  (hand-thrown  and  under-slung), 
suppression,  shooting  through  cover,  and  the  effects  of 
casualties. 

Although  the  treatment  of  casualties  is  beyond  the 
scope  of  this  paper,  it  is  notable  that  over  the  past  decade, 
virtual  reality  (VR),  which  in  this  case  includes  game-based 
applications,  is  starting  to  make  a  significant  impact  on 
behavioral  healthcare.  The  inaugural  issue  of  the  Journal  of 
Cyber  Therapy  and  Rehabilitation  (JCTR)  comprises  seven 
papers  that  describe  virtual  reality  applications  for  the 
treatments  of  psychological  problems  from  schizophrenia 
to  eating  disorders  (Weiderhold,  2008).  Of  particular  interest 
from  a  military  training  standpoint  is  a  paper  written  by 
Weiderhold  &  Weiderhold  (2008)  which  is  dedicated  to 
VR  applications  in  the  treatment  of  post  traumatic  stress 
disorder  (PTSD)  and  stress  indoctrination  training  (SIT). 

Australian  researchers  are  also  using  VR  integrated  with 
SG  technology  to  overcome  aviation  aircrewman  training 
deficiencies.  Carpenter  (2008)  describes  an  Australian 
Defence  Force  success  that  created  the  Aircrewman  Virtual 
Reality  Simulator  (AVRS).  This  simulator  provides  a  360 
degree  field  of  view  through  a  head  mounted  display  that 
is  stimulated  by  the  SG  VBS  2®.  Carpenter  explains  that 
the  AVRS  project  was  created  to  address  the  unacceptable 
failure  rates  for  aviation  air  crewman.  The  root  cause  was 
determined  to  be  insufficient  practice  in  appropriate 
scenarios  and  two  simulation  alternatives  were  evaluated 
to  meet  this  need.  They  compared  a  high  end  dedicated 
Military  Off-the-Shelf  (MOTS)  option,  which  was  low  risk 
but  expensive,  to  a  significantly  higher  risk  development 
around  VBS  2®  that  would  cost  considerably  less.  The 
project  was  approved  in  May,  2006  with  all  systems  being 
delivered  by  November,  2007.  Figure  3  shows  the  VBS  2® 
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based  AVRS  instructor  module  in  the  foreground  with  the 
trainee  module  in  the  background.  The  trainees  get  a  full 
360  degree  VR  view  through  the  headsets. 


Figure  3:  Aircrewman  Virtual  Reality  Simulator 
(Carpenter,  2008) 


Carpenter  reports  improved  performance  with  a 
substantially  improved  pass  rate,  higher  standards  and 
increased  throughput  as  primary  benefits  for  which  the 
system  was  created.  Additional  benefits  include  the 
opportunity  to  practice  in  more  realistic  scenarios  and  in  all 
types  of  weather.  Students  who  require  remedial  training 
have  the  opportunity  to  do  so  and,  as  seems  to  be  the  case 
when  cognitive  skills  are  improved  through  game-based 
techniques,  the  live  training  was  reported  to  have  become 
both  safer  and  more  effective. 

Effectiveness  and  Efficiency  of  Serious  Games 

The  efficiency  argument  for  serious  games  is  relatively 
easy  to  make  based  upon  cost  savings.  This  is  true  of 
simulation  in  general,  but  even  more  so  when  using 
commercial  off-the-shelf  (COTS)  software  compared  to 
relatively  more  expensive  purpose  built  military  simulations 
and  simulators  which  need  to  be  networked  together 
for  collective  training.  The  implicit  assumption  with  this 
argument,  however,  is  that  game-based  training  is  being 
used  to  replace  live  training  and  this  does  not  appear  to  be 
the  case.  Rather,  SG  training  is  increasingly  being  applied 
as  part  of  blended  training  programs  or  as  an  enhancement 
to  live  training.  What  the  SGs  may  be  replacing,  however, 
is  instructor  lead  classroom  learning  and  as  a  result,  may 


well  have  a  cost  increase  associated  with  their  adoption 
and  use.  Many  students  can  follow  a  single  PowerPoint 
presentation  on  how  to  perform  room  clearing  TTPs, 
however  for  a  squad  to  practice  and  learn  the  cognitive 
skills  in  the  SG  will  require  hardware  and  software  for  each 
trainee.  The  distinction  between  following  a  presentation 
and  learning  in  an  SG  is  intentional  because  the  SG  enables 
an  experience-based  approach  which  is  more  effective 
than  instructor  lead  PowerPoint  presentations  (Menaker 
et  al.,  2006).  However,  it  clearly  will  cost  more  than  classic 
classroom  instruction  which  is  very  efficient,  but  arguably 
not  very  effective  for  training  tactical  skills.  One  might 
observe  that  the  trend  to  increase  experience-based 
learning  through  an  increase  in  the  use  of  SGs  for  tactical 
training  is  really  a  return  to  the  training  paradigms  of  the 
cold-war  era  which  had  considerably  more  emphasis  on 
experience-based  live  field  training.  As  part  of  the  peace 
dividend,  resulting  from  the  end  of  the  Cold  War,  there  has 
been  a  significant  decrease  in  training  budgets  and,  as  a 
result,  considerably  less  "live"  experience-based  training. 
SGs  provide  a  cost-effective  alternative. 

An  excellent  illustration  of  the  relationship  between 
efficiency  and  effectiveness  of  game  based  training  is 
occurring  at  the  Canadian  Combat  Training  Centre  (CTC). 
CTC  comprises  several  specific  schools  and  is  currently 
experimenting  with  multiple  SGs  to  meet  dynamic  training 
requirements.  In  one  trial,  the  Armour  School  recently 
incorporated  VBS®  into  the  Troop  Warrant  Officer's  course. 
Hill  (2008)  provided  an  assessment  of  the  cost  savings 
(efficiency)  and  performance  improvements  (effectiveness) 
associated  with  increasing  the  amount  of  game-based 
training  in  successive  serials  of  the  Troop  Warrant  Officers 
course  conducted  in  2007.  In  the  first  serial  of  the  6  week 
course,  the  trainees  received  1  day  of  VBS®  training  and  5.5 
weeks  of  live  training  in  the  field.  Figure  4  shows  a  partial 
training  layout  including  a  tank  crew  (Leopard)  and  a 
surveillance  crew  (Coyote).  Figure  5  is  a  photograph  of  one 
of  the  workstations.  Based  on  the  success  with  VBS®  during 
the  first  serial,  the  training  staff  increased  the  VBS®  portion 
to  2.5  weeks  for  the  next  serial  and  decreased  live  training 
to  3  weeks. 

The  costs  of  the  second  serial  were  reduced  by 
approximately  33%  due  to  the  need  for  less  fuel,  food  and 
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DS/Observer 


Figure  4:  Partial  Layout  for  Armoured  Troop  Warrant  Officer 
Course  (Hill  2008) 


Figure  5:  VBS  2®  Workstations  in  the  Armour  School  Battle 
Lab  (Hill  2008) 


field  pay  compared  to  the  first  serial.  More  significant, 
however,  was  the  effect  on  performance  (effectiveness). 
Hill  measured  performance  based  upon  student  success 
rates  as  defined  by  the  proportion  of  students  that  pass  this 
demanding  course  and  the  number  of  traces  (live  battle 
runs)  needed  to  demonstrate  proficiency.  Table  1  presents 
the  results  from  three  consecutive  serials  of  this  course. 
Serial  0602  had  no  VBS®  training,  and  can  be  considered  a 
control  group.  All  three  serials  had  18  students. 


Serial 
0602 (No 
VBS®) 

Serial 
0701  (1 
day  VBS®) 

Serial  0702 
(2.5  weeks 
VBS®) 

%  pass  on  1st 
trace 

0 

30% 

67% 

%  pass  by  Vi  of 
traces 

61% 

72% 

100% 

%  pass  by  end 
of  course 

72% 

83% 

100% 

Table  1 :  Performance  Results  with  Increasing  Amounts 
of  Game-Based  Training  (Hill  2008) 

As  was  the  case  with  the  Australian  AVRS  example 
described  earlier,  the  game-based  training  resulted  in  a 
significant  improvement  during  the  field  portion  of  the 
course.  Serial  0702  was  reported  to  be  the  first  course  with 
a  100%  success  rate  which  was  achieved  in  all  cases  with 
only  two  live  traces  compared  to  the  six  traces  required 
to  achieve  an  83%  success  in  the  previous  serial.  In  his 
presentation  of  these  results,  Hill  pointed  out  that  there 
were  many  potential  confounds  associated  with  the 
improvements  in  performance  including,  for  example,  that 
there  were  different  instructors  for  each  course.  Even  so, 
the  impressive  improvement  in  performance  has  resulted  in 
CTC  considering  the  addition  of  SG  content  in  the  majority 
of  the  courses  they  deliver. 

Despite  this  clear  success,  Hill  and  others  emphasize  that 
the  cognitive  training  provided  by  game-based  simulation 
does  not  replace  the  need  for  live  training.  They  argue  that 
the  key  to  effective  tactical  training  is  to  develop  affordable, 
blended  training  solutions,  as  prescribed  by  Pringle  (2007), 
which  take  advantage  of  the  strengths  and  compensates 
for  weaknesses  of  all  of  the  training  modes  available. 

A  potential  strength  of  game-based  simulation  lies 
in  the  area  of  behavioral  or  affective  training.  Whereas 
cognitive  training  addresses  the  mental  skills  required  by 
trainees  to  develop  knowledge,  affective  training  deals 
more  with  their  emotional  state  and  ability  perform  under 
stress.  Although  there  is  a  growing  body  of  evidence 
supporting  the  effectiveness  of  game-based  training 
for  cognitive  learning,  there  are  relatively  few  studies 
exploring  the  affective  learning  aspects  of  game-based 
desktop  simulators.  A  recently  completed  three  year 
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Defence  Advanced  Research  Projects  Agency  (DARPA) 
sponsored  project  employed  physiological  monitoring 
during  simulation  training  and  testing  as  a  means  to 
assess  the  degree  of  affective  training  provided  by  the 
game  (Weiderhold  and  Weiderhold,  2006b).  The  overall 
objective  of  this  research  was  to  assess  the  effectiveness 
of  using  game-based  laptop  training  with  US  armed  forces 
personnel  participating  in  simulation  training  prior  to 
conducting  live  training.  Specifically,  the  investigation 
examined  the  effectiveness  of  desktop  training  simulators 
to  teach  tactical  and  trauma  care  skills,  to  practice  stress 
management  and  to  improve  performance  during  real-life 
combat  situations.  Nine  hundred  and  seventy  participants 
trained  in  a  virtual  combat  scenario  while  their  stress 
and  arousal  levels  were  monitored  through  noninvasive 
physiological  means.  A  control  group  did  not  receive 
game-based  training.  All  the  participants  were  then  tested 
in  a  live  version  of  the  same  combat  scenario  to  determine 
the  effectiveness  of  the  desktop  immersive  training.  One 
subgroup  of  210  United  States  Marine  Corp  (USMC) 
soldiers  was  observed  during  an  1 1-day  training  program. 
Ninety  of  the  210  subjects  received  desktop  immersive 
training  prior  to  live  training  and  their  performance  was 
compared  to  the  remaining  120  subjects  who  did  not 
receive  desktop  training.  Although  the  scenario  in  this 
example  (proper  identification  and  Breach  techniques  for 
a  shoot  house),  was  significantly  different  from  the  open 
terrain  armoured  traces  described  in  Hill's  presentation, 
the  results  were  very  similar.  All  personnel  who  had  the 
opportunity  to  perform  the  scenarios  in  the  game  prior  to 
the  live  training  were  assessed  to  be  100%  accurate  on  all 
runs  whereas  the  control  group  (no  game-based  rehearsal) 
was  only  80%  accurate  80%  of  the  time.  Furthermore,  it 
was  reported  that  the  game-based  rehearsals  resulted  in 
improved  spatial  awareness  and  the  trainees  completed 
their  tasks  more  quickly  with  less  need  for  communication 
as  the  team  skills  had  been  improved  during  the  rehearsal. 
The  desktop  immersive  trained  personnel  performed 
better  at  spatial  awareness  within  the  shoot  house,  moved 
more  quickly  entering  the  shoot  house  and  required  less 
communication  than  the  control  group,  as  each  person 
was  able  to  anticipate  the  other/team  movements,  in 
comparison  to  the  control  group  that  took  an  average  of  6 
seconds  longer  to  perform  the  same  tasks. 


The  study  went  into  further  details  exploring  more 
affective  aspects  for  different  groups  of  US  personnel 
and  came  to  the  conclusion  that  affective  and  cognitive 
learning  in  desktop  immersive  trained  groups  helped  them 
out  perform  in  all  tasks  compared  to  those  that  had  not 
received  the  training.  Weiderhold  and  Weiderhold,  (2008) 
explain  this  phenomenon  as  stress  inoculation  training 
(SIT)  and  argue  that  the  benefits  described  above  are  due 
in  part  to  the  experience  gained  in  the  game  and  how  that 
reduces  the  stress  of  performing  in  the  live  equivalent. 
This  effect  has  been  so  pronounced,  that  the  success 
demonstrated  with  these  military  examples  has  resulted  in 
similar  research  efforts  to  provide  SIT  for  medical  personnel, 
the  US  Coast  Guard  and  SWAT  teams  (Weiderhold  and 
Weiderhold,  2008). 

Another  good  practical  reason  to  use  simulation  for 
training  is  if  the  actual  equipment  is  7000  miles  away.  In  an 
excellent  blended  learning  example,  O'Bea  and  Beacham 
(2008)  described  how  three  types  of  simulation  are  being 
used  to  train  US  soldiers  on  route  clearing  operations  prior 
to  deployment  in  IRAQ.  The  motivation  for  this  work  was 
based  upon  the  fact  that  the  primary  equipments  used 
for  clearing  improvised  explosive  devices  (lEDs)  are  being 
fielded  directly  into  theatre  without  any  systems  in  the 
US  to  support  training.  Operators  do  not  see  the  actual 
equipment  until  they  are  deployed.  To  compensate  for 
this  gap,  O'Bea  and  Beacham  (2008)  described  how  the 
combination  of  a  "live"  part-task  trainer,  a  virtual  reality 
system  trainer  and  a  game-based  convoy  trainer  have 
been  employed  to  get  the  Army  Engineer  Clearance 
Company  ready  for  deployment.  The  part-task  trainer  is 
a  hardware-based  simulator  with  identical  controls  and 
performance  characteristics  to  the  mine  clearing  arm  on 
the  Buffalo  that  enables  the  operators  to  develop  their 
psycho-motor  skills  as  operators  of  the  equipment.  The 
Virtual  Route  Clearance  Trainer  (VRCT)  is  combined  with 
classroom  instruction  to  allow  detachments  to  train  as  a 
team  on  the  latest  mine  clearing  TTPs  and  is  reconfigurable 
to  represent  three  different  mine  clearing  systems.  The 
training  culminates  with  a  game-based  convoy  scenario 
that  employs  DARWARS  Ambush  in  which  the  clearance 
company  detachments  take  up  their  role  as  part  of  a 
convoy  allowing  the  full  convoy  team  to  train  together  in 
realistic  scenarios  that  are  tailored  to  meet  specific  threats 
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expected  to  be  encountered  during  operations.  Figure  6 
shows  the  DARWARS  Ambush  visual  models  of  three  of  the 
clearance  company's  vehicles  including  the  Buffalo  with 
the  articulated  arm  extended. 


Figure  6:  DARWARS  Ambush  Screenshot  (US  Army  Photo, 
O'Bea  and  Beacham,  2008) 


The  mine  clearing  example  and  its  use  of  a  part-task 
trainer  serves  as  a  good  reminder  that  Desktop  game-based 
training  is  not  well  suited  forthe  psycho-motor  aspects.  This 
is  the  one  area  in  which  certain  negative  learning  aspects 
may  come  into  play.  To  overcome  this,  low  level  mock  ups 
and  stations  must  be  created  that  replicate  the  training 
that  is  desired  and  do  not  cause  participants  to  achieve 
proficiency  in  an  aspect  of  training  that  is  not  realistic. 
However,  desktop  game-based  simulations  can  provide 
leaders  and  team  members  the  ability  to  effectively  practice 
the  cognitive  and  decision  making  skills  that  they  will  need 
in  real  world  situations.  Knerr  (2007)  states  that  through 
focused,  repetitive,  deliberate  practice  with  feedback  based 
on  performance,  this  is  an  ideal  and  effective  method  for 
training.  What  the  examples  above  serve  to  illustrate  is  that 
this  training  should  not  replace  live  training,  but  rather  SGs 
can  make  live  training  much  more  effective  and  efficient 
with  soldiers  meeting  and  even  exceeding  standards  more 
quickly.  The  experience-based  cognitive  and  affective 
learning  provided  by  well  conducted  game-based  training 
therefore,  can  make  an  overall  blended  approach  both 
more  effective  and  efficient. 

What  are  the  Technological  Limits  of  Serious  Games? 

In  examining  the  last  question,  two  serials  of  a  Canadian 
aviation  exercise,  Winged  Warrior  will  be  considered. 


Exercise  Winged  Warrior  has  traditionally  been  a  live 
exercise  intended  to  test  tactical  helicopter  pilots  in  their 
role  as  aviation  mission  commanders  during  the  planning 
end  execution  of  complex  missions.  It  serves  as  an 
excellent  example  to  examine  technical  limits  as  tactical 
aviation  is  arguably  the  most  difficult  (military)  case  for  an 
SG  considering  terrain  models  and  graphics  performance 
requirements.  In  addition  to  being  relatively  fast  movers 
capable  of  covering  large  geographical  areas,  helicopters 
fly  at  low  altitude  demanding  a  high  degree  of  visual  detail. 
Aircraft  flying  fast  and  high  can  get  by  with  a  low-resolution 
picture  draped  over  a  low  fidelity  Digital  Terrain  Elevation 
Data  (DTED)  skin.  This  is  not  the  case  for  aviation  missions 
where  pilots  may  often  fly  nap  of  the  earth  and  would 
be  especially  limiting  in  exercises  like  Winged  Warrior 
that  include  infantry  units  being  supported  by  aviation. 
Winged  Warrior  also  requires  that  the  SG  be  federated  with 
other  simulations  in  order  to  meet  a  broad  set  of  training 
objectives. 

Typical  aviation  missions  executed  during  the  exercise 
included: 

•  Reconnaissance  and  surveillance 

•  Direction  and  control  of  fire 

•  Provision  of  fire  support 

•  Combat  airlift/tactical  transport 

•  Logistical  transport 

•  Communications  support 

Roman  and  Brown  (2007)  describe  that  the  cost  of  the 
live  exercise  had  become  too  great  resulting  in  the  creation 
of  a  game-based  equivalent  using  a  combination  of  the  SG 
Steal  Beasts®  and  the  Joint  Conflict  and  Tactical  Simulation 
(JCATS)  constructive  simulation  with  the  first  serial 
occurring  in  2006.  Although  the  2006  version  of  Winged 
Warrior  was  a  tremendous  success  as  the  first  SG  application 
for  command  and  staff  training  at  the  Directorate  of  Land 
Synthetic  Environments,  the  sponsors  of  the  2007  version 
of  the  exercise  wanted  to  overcome  the  following  shortfalls 
from  the  previous  year: 

•  Communications  simulation  limitations 

•  Lack  of  night  operations 

•  No  civilian  personnel  or  vehicles 
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•  No  navigation  instruments  or  electronic  defence 
measures 

•  No  door  guns 

•  No  control  over  weather 

•  No  fast  air 

•  Poor  overview  of  entire  battle-space 

There  was  also  a  strong  desire  on  the  part  of  the  Air  Force 
to  expand  the  scope  of  the  exercise  to  include: 

•  Improving  the  in-air  Command  and  Control 
presentation 

•  Expanding  the  training  audience  to  include  a 
Helicopter  Squadron's  Battle  Staff 

•  The  addition  of  distributed  simulators. 

Winged  Warrior  07  was  executed  during  the  period  22- 
31  Oct  2007  at  CFB  Valcartier,  just  outside  of  Quebec  City. 
Figure  7  shows  how  the  Area  Simulation  Centre,  housed  in  a 
former  indoor  rifle  range,  was  configured  for  the  training. 

Winged  Warrior  was  an  unclassified  exercise  set  in 
simulated  Afghanistan  using  typical  aviation  missions  on 
geo-specific  terrain.  See  Figure  8  for  an  out  the  window 
view  for  the  helicopter  pilot  in  VBS  2®.  Threats  were 
typical  and  varied  from  the  very  simple  to  the  extremely 
complex  and  included  both  civilian  and  non-combatants. 
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Figure  7:  Exercise  Winged  Warrior  2007  Layout 


To  overcome  the  shortfalls  from  the  2006  version  of  the 
exercise,  a  simulation  federation  was  formed  with  IEEE  1516 
(HLA)  forming  the  basis  for  the  link  to  three  geographically 
dispersed  exercise  locations  (Valcartier,  Ottawa  and 
Kingston).  IEEE  1278  (DIS)  was  used  to  link  VBS  2®  with 
JCATS.  The  following  simulation  components  formed  the 
federation: 

•  VBS  2® 

•  JCATS 

•  Raptor  Simulator 

•  ASTi®  simulated  Radio 

VBS  2®  was  selected  as  the  primary  visual  simulation 
for  the  exercise.  The  majority  of  objects  on  the  simulated 
battlefield  were  generated  through  VBS  2®  with  JCATS 
providing  a  larger  background  air  picture  and  the  Raptor 
simulator  providing  a  single  virtual  platform  for  the 
exercise.  Given  the  size  of  the  terrain  and  the  number  of 
objects  that  had  to  be  represented,  the  capability  of  VBS  2® 
was  considerably  stretched.  Indeed  it  was  well  outside  the 
nominal  bounds  of  what  this  game  engine  was  designed  to 
be  able  to  handle  (ground  based  personnel  and  vehicles). 
This  was  the  first  military  exercise  that  employed  VBS  2®  for 
a  tactical  aviation  primary  training  audience.  Despite  these 
challenges,  VBS  2®  performed  very  well,  meeting  most  of 
the  pre-exercise  training  requirements.  Game  engines 
clearly  have  come  a  very  long  way.  The 
ability  to  process  and  display  a  very  large 
terrain  file  with  a  high  degree  of  detail 
and  lots  of  simulation  entities  speaks  to 
a  maturity  of  this  technology.  What  was 
achieved  on  normal  computer  hardware 
using  an  adapted  first-person  shooter 
game  for  this  exercise  simply  could  not 
have  been  achieved  two  years  ago. 


The  VBS  2®  tool  suite  that  supports 
terrain  and  scenario  generation  is  due 
for  a  major  upgrade  in  2008.  These 
tools  were  in  their  infancy  during  the 
lead  up  period  to  Winged  Warrior.  This 
complicated  the  construction  of  the 
terrain  model  for  the  exercise  as  this 
was  the  most  complicated  user  created 
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terrain  model  to  date.  Bohemia  Interactive  was  extremely 
helpful  in  assisting  in  the  use  of  these  tools.  Many  of 
the  observations  made  and  lessons  learned  have  been 
integrated  in  the  new  versions  of  these  tools  that  will  be 
sold  as  VBS  2  VTK®  (Butcher,  Johnson  and  Morrison,  2008) 
which  will  comprise  both  the  game  software,  the  VBS 
Tool  Kit  (VTK)  to  allow  user  modification  and  the  software 
necessary  to  link  VBS  2®  with  virtual  or  constructive 
simulations  using  HLA  or  DIS  as  appropriate. 

The  inclusion  of  a  general-purpose  constructive 
simulation  proved  to  be  a  very  useful  component  to  the 
synthetic  environment.  From  it,  the  overall  air  picture 
was  easily  generated  and  able  to  stimulate  the  secondary 
training  audience  from  the  helicopter  squadron  command 
post.  This  change  from  the  2006  version  of  the  exercise 
was  so  successful  that  an  even  more  advanced  VBS  2® 
and  JCATS  synthetic  environment  was  created.  It  is  being 
used  to  support  high  readiness  training  of  helicopter  units 
potentially  deploying  to  Afghanistan  in  2008  and  for  future 
iterations  of  Winged  Warrior. 


Figure  8:  Pilot  view  in  VBS  2® 


To  date,  the  Winged  Warrior  series  of  exercises,  arguably 
among  the  most  demanding  scenarios  for  SGs,  have  yet  to 
be  constrained  by  technical  limitations.  Looking  forward, 
the  hardware  and  software  will  only  improve.  Bigger  terrain 
tiles,  more  terrain  objects,  better  visual  effects  and  more 
entities  will  be  able  to  be  displayed  and  natural  market 
forces  will  result  in  game  companies  continuing  to  compete 
through  improved  capabilities  which  will  in  turn  allow  the 


generation  of  richer  and  richer  visual  environments  and 
ultimately  better  and  better  training  of  this  type. 

Summary 

This  paper  set  out  to  address  three  questions  regarding 
the  use  of  serious  game  technology: 

•  What  tactical  training  requirements  are  serious  games 
best  suited  to  meeting? 

•  How  effective  and  efficient  are  they  at  meeting  those 
requirements? 

•  What  are  the  technical  and  pedagogical  limits 
associated  with  there  use? 

Several  specific  examples  have  been  provided  that 
address  the  first  question.  It  would  be  presumptuous, 
however,  to  assume  that  a  list  of  specific  requirements 
can  be  provided  since  the  technology  continues  to  evolve, 
new  requirements  continue  to  emerge  and  innovative 
researchers  and  trainers  will  continue  to  discover  new  uses 
over  time.  It  is  possible,  however,  to  generalize  from  the 
examples  provided  to  emphasize  that  SGs  are  providing  a 
cost-effective  means  to  provide  experience-based  learning 
with  emphasis  on  cognitive  and  increasingly  affective 
training  domains.  War  fighters  will  not  develop  the  expert 
psycho-motor  skills  they  need  to  effectively  employ  their 
weapon  systems  using  game-based  training.  However, 
once  the  team  of  experts  in  various  weapon  systems  is 
created,  SG  technology  affords  trainers  the  opportunity  to 
turn  them  into  an  expert  team  capable  of  communicating 
effectively  with  the  cognitive  skills  they  need  to  effectively 
operate  as  teams. 

The  body  of  evidence  supporting  the  effectiveness  of  SG 
training  solutions  is  starting  to  build  and  examples  from 
the  UK,  Canada,  the  US  and  Australia  have  been  provided. 
In  all  cases,  the  examples  provided  support  the  theoretical 
arguments  that  SG  technology  is  most  effective  as  part  of 
a  blended  training  solution  that  takes  advantage  of  the 
strengths  of  the  available  training  tools  and  processes. 
One  consistent  result  is  that  using  the  SG  prior  to  live 
training  makes  live  training  more  effective  and  efficient. 
In  the  British,  Canadian  and  USMC  examples  provided 
above,  pass  rates  and  performance  conducted  during  live 
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training  were  significantly  improved  when  rehearsals  were 
conducted  using  the  SG  compared  to  those  that  did  not  get 
the  SG  practice.  In  some  cases,  simulation  may  be  the  only 
training  option  as  the  trend  to  field  the  latest  equipment 
directly  into  theatre  results  in  a  complete  lack  of  training 
capability  at  the  home  station.  Employing  this  equipment 
during  collective  training  conducted  in  an  SG  as  described 
above  for  the  US  Army  Engineers  Mine  Clearance  Company 
is  arguably  the  only  way  to  prepare  troops  to  participate 
in  combined  arms  teams  using  equipment  they  will  not 
actually  touch  until  they  arrive  in  theatre. 

In  addressing  the  last  question,  the  Winged  Warrior 
experience  clearly  indicates  that  technology  is  not  a  limiting 
factor  in  terms  of  SG  support  to  training  exercises.  Terrain 
database  generation  is  improving  rapidly,  the  games  can  be 
federated  with  other  virtual  and  constructive  simulations  to 
meet  a  broad  range  of  training  needs  and  technology  and 
market  forces  guarantee  that  the  technology  will  continue 
to  improve  over  time. 

Conclusions 

Serious  Game  technology  is  an  effective  means  to  meet 
a  wide  variety  of  tactical  training  requirements  and  is 
particularly  well  suited  to  developing  the  cognitive  skills 
necessary  to  turn  a  team  of  experts  into  an  expert  team.  To 
take  advantage  of  this  capability,  SGs  need  to  be  included  as 
part  of  blended  training  solutions  that  can  cost-effectively 
meet  the  psycho-motor  and  affective  training  requirements 
as  well.  In  stand-alone  mode,  the  games  appear  to  be  best 
suited  from  individual  up  to  company  level,  however,  when 
used  in  combination  with  other  virtual  and  constructive 
simulations  may  well  assist  in  the  collective  training  of 
larger  groups. 

Returning  to  the  original  question  posed  in  the  title 
of  this  paper:  Games  -  just  how  serious  are  they?  The 
answer  is  very  serious.  So  serious  in  fact  that  it  may  not 
be  appropriate  to  call  them  games  at  all.  Although  not 
described  in  this  paper,  some  training  establishments  may 
havedifficultyacceptinggamesascredibletrainingaids.  We 
could  of  course  call  them  wargames,  or  desktop  immersive 
trainers,  however  the  authors  of  this  paper  believe  that  it 
is  very  natural  for  human  beings  to  learn  through  play  and 


the  use  of  games,  albeit  for  some  very  serious  purposes,  is 
an  obvious  and  natural  fit  for  tactical  training. 
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Introduction:  Problems  with  Modeling  Human 
Behavior 

There'sa  long  way  to  go  before  the  Department  of  Defense 
can  model  human  behavior  well  enough  to  support  the 
analysis  of  irregular  warfare  (IW),  according  to  two  books 
published  this  year  by  the  National  Research  Council  (NRC). 
The  nonprofit  institution,  which  exists  to  provide  technical 
advice  to  the  federal  government,  published  "Behavioral 
Modeling  &  Simulation"  in  July.  Earlier  this  year,  the  NRC 
published  "Human  Behavior  in  Military  Contexts,"  which 
primarily  addresses  training  but  still  considers  analysis. 

The  subject  of  the  books  is  a  vital  one,  because  it's  likely 
Marines  will  face  IW  for  the  foreseeable  future.  The  books 
command  more  attention  than  typical  documents  because 
the  NRC  produced  them.  The  NRC's  authors  are  respected 
scholars,  and  the  NRC  emphasizes  grand  research  issues 
rather  than  specific  problems.  To  see  two  books  come  out 
in  one  year  on  a  specific  military  need  is  unusual  good 
fortune. 

Both  books  revolve  around  understanding  human 
behavior,  which  makes  them  relevant  to  analysts  who  have 
been  exploring  how  to  apply  social  sciences  to  the  study  of 
IW.  Analysts  will  find  the  most  value  in  "Behavioral  Modeling 
&  Simulation,"  which  has  a  number  of  recommendations, 
including  these: 

1.  Improve  the  theory  underlying  models  of  human 
behavior.  Current  models  typically  have  inadequate 
theory. 

2.  Be  ready  to  account  for  and  accept  uncertainty 
and  change. There  are  important  things  we  won't  be 
able  to  know.  And  there  are  important  things  we  will 
know  initially  but  won't  understand  once  population 
behavior  changes. 

3.  Research  how  to  collect  data  for  behavioral 
models.  Getting  data  suitable  for  behavioral 
models  is  very  difficult,  and  modelers  are  prone 
to  misunderstanding  the  relationships  between 
behavioral  data  and  behavioral  models. 

4.  Make  sure  that  a  federation  of  behavioral  models 


rests  upon  compatible  assumptions.  All  the  parts  in 
your  system  diagram  may  appear  to  make  a  coherent 
whole.  And  the  connections  may  permit  each  node 
in  the  network  to  "talk"  to  the  others.  However,  where 
human  behavior  is  concerned,  different  researchers 
can  have  radically  different  assumptions.  Failure  to 
maintain  harmony  with  the  assumptions  is  likely  to 
produce  meaningless  results. 

5.  In  the  Verification,  Validation,  &  Accreditation 
(VV&A)  process,  there  needs  to  be  a  special 
method  of  validation  devised  with  behavioral 
models  in  mind.  You  can't  use  the  validation 
techniques  you  would  use  for  a  physics-based  model 
because  you  can't  test  against  the  real  world  the  way 
you  can  with  laminar  fluid  flow  or  falling  bodies. 

Social  Sciences 

It  may  sound  a  bit  naive  to  say  you're  "reviewing  social 
science,"  given  the  number  and  complexity  of  disciplines 
that  qualify  as  social  sciences,  observed  Dr.  Jim  Stevens 
of  OSD  PA&E.  Nevertheless,  he  continued,  it's  necessary  to 
develop  a  basic  understanding  of  the  relevant  principles 
from  the  social  sciences  before  attempting  to  create 
behavioral  models. 

This  is  reasonable  advice  for  Marine  analysts  to  follow 
as  they  contemplate  what  the  NRC  says  about  modeling 
human  behavior.  Analysts  might  see  potential  in  such 
models,  but  right  now  the  requisite  knowledge  isn't  ready 
for  prime  time. 

Definitions 

The  social  sciences  and  the  military  are  notorious  for 
having  revolving  doors  of  misleadingly  precise  jargon. 
Throughout  this  paper,  terms  such  as  "behavior",  "social 
sciences",  and  "irregular  warfare"  are  used  loosely.  It's 
difficult  enough  to  summarize  the  essence  of  what  experts 
are  doing  in  these  fields.  It's  inconceivable  that  a  brief  paper 
can  standardize  nomenclature  for  DOD  or  a  significant 
chunk  of  academia. 

Even  if  we  set  aside  the  social  sciences  and  focus  strictly 
on  DOD  labels,  we  witness  unsatisfactory  descriptions  that 
sometimes  resemble  shorthand  for  incomplete  laundry 
lists.  This  array  of  acronyms  and  abbreviations  includes: 
IW;  Counterinsurgency  (COIN);  Counterterrorism  (CT); 
Diplomatic  Informational  Military  Economic  Financial 
Intelligence  Law  Enforcement  (DIMEFIL);  Political  Military 
Economic  Social  Infrastructural  and  Informational  (PMESII); 
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Military  Operations  OtherThan  War  (MOOTW);  and  Security 
Stability  Transition  and  Reconstruction  (SSTR). 

This  paper's  refusal  to  be  strict  about  terminology 
stems  from  IW's  "squishy"  nature.  As  an  upcoming  RAND 
report  will  note,  even  a  basic  line  between  terrorism  and 
insurgency  is  a  blurry  line. 

Marine  Operational  Concept 

TheCommandantoftheMarineCorps,Gen.JamesConway, 
released  "The  Long  War"  in  the  past  year.  He  described 
his  36-page  document  as  an  "operational  employment 
concept."  He  outlined  a  variety  of  problems  Marines  must 
be  ready  to  face,  and  each  one  of  these  problems  would 
be  easier  to  tackle  with  a  strong  understanding  of  human 
behavior. These  problems  include: 

•  Various  forms  of  IW,e.g.,  terrorism 

•  Ideological  or  religious  extremism 

•  Ethnic  polarization 

•  Poorly  governed  or  ungoverned  regions 

•  Weak  economies 

•  Poverty 

•  Health  crises 

•  Competition  for  resources  such  as  water  or  energy 

•  Demographic  problems,  e.g., "youth  bulge" 

•  Environmental  problems,  e.g.,  a  tsunami 

•  Crime 

Each  of  the  problems  Gen.  Conway  highlights  is  an 
asymmetric  challenge  -  what  the  blue  side  does  usually 
differs  from  what  the  red  side  does.  This  is  challenging 
for  the  analyst.  It  means  having  one  set  of  metrics  to 
judge  how  effective  blue  is,  and  another  set  of  metrics  to 
judge  how  effective  red  is.  As  a  group  of  panelists  recently 
assembled  by  RAND  observed,  the  conflict  between  the 
US  and  Osama  Bin  Laden  vividly  illustrates  an  asymmetric 
conflict.  He  makes  small  investments  in  attempts  to  cost  us 
lots  of  money.  We  make  large  investments  in  attempts  to 
kill  him,  kill  his  minions,  and  win  over  fence-sitters.  Either 
way,  human  behavior  plays  a  large  role.  Osama  hopes  the 
US  public  will  grow  tired  of  war.  We  hope  fence-sitters  will 
choose  peace  over  extremism. 

Counterinsurgency 

Influencing  human  behavior  is  at  the  root  of  most  ideas 
about  counterinsurgency.  By  most  counterinsurgency 
theories,  blue  and  red  may  use  entirely  different  methods, 


but  each  seeks  to  bring  the  population  over  to  its  side. 
The  population  may  act  out  of  fear,  love,  or  some  other 
consideration,  but  insurgents  and  counterinsurgents 
usually  stake  their  hopes  for  success  on  the  behavior  of  the 
population.  This  raises  some  questions  for  the  analyst,  such 
as: 

1. How  do  you  evaluate  the  population's  opinions  and 
behaviors? 

2.  If  something  affects  the  population's  behavior,  how 
long  does  that  effect  hold  up? 

3.  What,  exactly,  is  the  "population"? 

The  questions  that  make  counterinsurgency  so  difficult 
were  at  the  heart  of  a  recent  discussion  hosted  by  the 
Center  for  Naval  Analyses.  The  toughest  questions  were 
asked  by  Dr.  David  Kilcullen,  considered  by  many  the 
world's  leading  expert  on  counterinsurgency.  He  practiced 
counterinsurgency  in  Australia's  army,  he  served  a  year  in 
Iraq  as  Gen.  David  Petraeus's  counterinsurgency  adviser, 
and  he  currently  serves  as  the  counterinsurgency  adviser 
for  Secretary  of  State  Condoleezza  Rice. 

"Is  counterinsurgency  a  strategy?"  he  asked.  "Is  it  an 
operational  art  form  ...  or  is  it  just,  in  fact,  a  set  of  tactics?" 
This  is  a  "question  a  lot  of  us  are  worrying  about  in  the 
policy  community,"  he  said. 

With  so  much  uncertainty  regarding  the  most  basic 
questions,  analysts  must  take  great  care  as  they  dig  into 
trickier  questions. 

While  physicists,  chemists,  and  engineers  can  build 
steadily  on  the  observations  of  peers  and  predecessors, 
analysts  of  counterinsurgency  have  a  harder  time  working 
with  lessons  from  the  past.  There  are  always  nagging 
questions  about  whether  the  things  we've  seen  before 
are  different  from  the  things  we'll  see  next.  As  Dr.  Kilcullen 
pointed  out,  you  see  this  problem  when  people  try  to  draw 
extensively  from  successful  British  counterinsurgencies  in 
Malaya  and  Northern  Ireland.  There  may  be  a  "fundamental 
difference  between  conducting  counterinsurgency 
in  your  own  country  or  in  a  colony  versus  conducting 
counterinsurgency  in  somebody  else's  country,"  Dr. 
Kilcullen  said.  The  key  difference,  he  continued,  is 
psychological.  When  you  go  to  a  foreign  land  and  conduct 
a  counterinsurgency,  "everyone  knows  you're  leaving" 
sooner  or  later. 

Uncertainty 

While  policy  makers  and  operators  must  deal  with 
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uncertainty  regarding  strategy,  doctrine,  and  tactics, 
analysts  must  deal  with  uncertainty  regarding  behavioral 
theories  and  behavioral  data.  As  the  NRC  observed  in 
"Behavioral  Modeling  &  Simulation,"  you  typically  lack  the 
data  you  need  to  model  something  such  as  a  terrorist 
network.  You  probably  won't  know  how  closely  your 
model  matches  the  part  of  the  world  you're  examining.  You 
might  not  know  if  the  scope  of  your  model  is  appropriate 
for  the  scope  of  the  problem.  Lastly,  some  of  your  initial 
understanding  will  diminish  as  people  change  their 
behavior  and  their  institutions. 

Theory 

Theories  about  human  behavior  tend  to  be  neglected, 
the  NRC  argues  in  "Behavioral  Modeling  &  Simulation."This 
happens  in  academia  as  well  as  in  DOD  because  funding 
favors  "doing"  rather  than  "thinking."The  unfortunate  result 
is  bad  conceptual  models.  While  modeling  has  the  allure  of 
being  tangible,  "strong  theory  and  a  clear  specification  of 
purpose"  are  far  more  important  to  understanding  human 
behavior  than  a  model  with  lots  of  "features  and  variables," 
the  NRC  said. 

This  problem  is  often  apparent  at  both  DOD  and 
academic  conferences.  If  the  subject  is  modeling  human 
behavior,  it's  common  to  find  more  modeling  experts 
than  behavior  experts.  As  one  person  observed  at  a  recent 
conference  hosted  by  National  Defense  University,  we 
don't  begin  our  analysis  by  asking  the  most  basic  question: 
"What's  the  problem?"  As  someone  else  observed,  it's  as  if 
we  wish  to  skip  over  the  problem  statement,  skip  over  the 
application  of  theory,  and  skip  over  the  careful  collection 
of  data.  "We  rush  to  quantify  what  we  don't  understand,"  he 
continued,  and  we  behave  as  if  the  act  of  making  a  model 
can  somehow  "ratify"  the  ideas  the  model  represents. 

It's  unsettling  that  behavioral  models  still  have  weak 
theory.  The  NRC  identified  this  problem  in  1998  when  it 
published  "Modeling  Human  and  Organizational  Behavior." 

While  behavioral  theories  are  abstract  and  difficult  to 
pin  down,  they  are  an  unavoidable  issue  in  the  practice 
of  IW.  For  example,  in  "The  Long  War,"  Gen.  Conway  gives 
an  example  in  which  the  Marines  may  find  themselves 
providing  "visible  forward  presence"  to  provide  a  country 
with  "economic  stabilization."  An  analyst  supporting  this 
type  of  endeavor  has  to  understand  the  relationships 
between  security,  foreign  military  forces,  and  the  economy 
of  a  host  nation.  These  are  theoretical  matters. 

Similarly,  Gen.  Conway  wrote  that  the  Marines  may  find 


themselves  in  a  country  suffering  from  economic  decline, 
poverty,  corruption,  epidemics,  and  ethnic  strife.  The 
analyst  assessing  which  problem  to  attack  first  must  have  a 
sound  theory  to  guide  him. 

An  analyst  can't  pull  theories  from  a  database  and 
plug  them  into  a  model.  In  "Human  Behavior  in  Military 
Contexts,"  the  NRC  argues  that  understanding  behavior 
is  difficult,  and  answers  vary  from  region  to  region.  For 
example,  the  NRC  says  you  would  need  psychologists, 
economists,  neuroscientists,  and  anthropologists  to 
understand  "cultural  variation  in  basic  processes,  such  as 
trust,  reciprocity,  cooperation  and  competition." 

"It's  the  Data,  Stupid" 

The  behavioral  data  that  can  be  collected  is  often 
inconsistent  with  the  data  requirements  for  a  behavioral 
model,  the  NRC  lamented  in  "Behavioral  Modeling  & 
Simulation."  While  certain  problems  with  behavioral  data 
will  always  exist,  others  are  correctable,  the  NRC  said.  In 
order  to  get  past  garbage-in  and  garbage-out,  modelers 
need  to  share  more  of  their  responsibilities  with  Subject 
Matter  Experts.  Experts  in  social  sciences  and  experts  in 
counterinsurgency  need  a  bigger  role  in  creating  models, 
identifying  variables,  and  choosing  assumptions. 

The  NRC  doubts  there  are  universal  laws  of  human 
behavior  and  social  structure.  Even  if  such  laws  existed,  the 
NRC  adds,  it  would  be  difficult  to  codify  them  and  compile 
empirical  data,  and  it  would  be  "unlikely  that  they  could 
be  represented  as  closed-form  equations."  In  other  words, 
there  are  no  quick  fixes  for  behavioral  data.  Collection  and 
use  of  such  data  require  care  and  judgment. 

Social  scientists  wrestle  with  the  problems  of  behavioral 
data  frequently.  They  know  that  if  you  interview  someone 
about  what  he  values,  he  might  say  one  thing  and  even 
mean  what  he  says.  But  if  you  observe  him,  you  might  see 
that  he  the  way  he  spends  his  time  doesn't  square  with 
what  he  says  is  important  to  him.  Alternatively,  you  might 
collect  data  regarding  some  sort  of  social  change,  but  it 
might  be  very  difficult  to  tell  whether  the  data  reflects  the 
actual  mechanism  of  change  or  merely  reflects  a  superficial 
manifestation  of  the  change.  As  yet  another  example,  you 
might  have  a  lot  of  data  about  a  population  at  peace,  but 
the  story  those  data  tell  might  say  nothing  about  how  the 
population  would  behave  during  a  war  or  a  disaster. 

A  social  scientist  can  have  a  lot  of  data  and  still  lack  the 
data  he  needs.  There  are  many  different  types  of  problems 
-  economic,  political,  religious,  ethnic,  etc.  These  problems 
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Figure  1 :  Regions  of  potential  instability  (from  "The  Long  War"). 


might  exist  in  a  variety  of  combinations.  There  are  also 
many  countries  that  are  potential  battlegrounds,  and  many 
of  these  countries  look  quite  different  as  you  move  from 
one  region  to  another.  General  Conway's  "The  Long  War" 
includes  a  map  highlighting  regions  of  actual  or  potential 
instability.  The  circled  region  of  the  map  gives  analysts 
more  than  1 00  countries  to  consider  (see  Figure  1  below). 

The  data  challenges  shift  as  you  move  from  one  country 
to  the  next  and  from  one  type  of  IW  operation  to  another. 
As  an  example  of  an  IW  operation,  Gen.  Conway  says  the 
Marines  might  find  themselves  working  with  Liberia's 
"ethnically  fractious  military"  in  order  to  make  it  more 
professional.  As  another  example,  Gen.  Conway  says  the 
Marines  might  find  themselves  in  Kenya,  working  with 
Non-Government  Organizations  to  deliver  humanitarian 
relief  following  a  severe  drought.  Both  operations  might  be 
characterized  as  IW  operations  supporting  the  Global  War 
on  Terror.  But  an  analyst  probably  would  need  two  entirely 
different  sets  of  data  to  study  these  problems.  It's  likely  that 
none  of  the  data  relevant  to  the  Liberia  operation  would  be 
useful  for  the  Kenya  operation,  and  vice  versa. 

The  Army's  Human  Terrain  System  currently  shows 
how  easy  it  is  for  social  science  data  from  IW  to  become 
a  collection  of  stovepipes.  Social  scientists  and  supporting 
personnel  deploy  to  Iraq  and  Afghanistan  in  teams.  They 
typically  support  battalion  commanders.  In  accordance 
with  counterinsurgency  doctrine,  these  commanders 
function  with  decentralized  authority.  Consequently, 
the  Human  Terrain  teams  aren't  coordinated  in  their 
data  collection.  One  battalion  commander  might  use  his 
Human  Terrain  team  to  compile  biographical  narratives  on 
a  village's  leaders.  Another  battalion  commander  might 
use  his  Human  Terrain  team  to  make  population  estimates 
and  keep  notes  on  the  economy.  The  result  is  plenty  of 
data,  but  very  little  in  the  way  of  systematic  data  collection 


or  data  standards. 

Behavior 

The  behavioral  questions  interesting  DOD  have  changed 
dramatically  in  10  years.  When  the  NRC  published 
"Modeling  Human  and  Organizational  Behavior"  in  1998, 
the  focus  was  conventional.  Typical  behavioral  questions 
involved  how  to  train  submarine  crews,  how  to  model  the 
pilot  of  an  aircraft,  and  how  to  model  the  commander  of  a 
tank  unit. 

If  it  was  difficult  to  model  members  of  a  military 
following  their  doctrine,  then  it's  bewildering  to  model 
civilians  as  they  determine  their  goals,  make  plans,  learn, 
make  decisions,  and  so  on.  And  cognitive  processes  aren't 
straightforward,  mechanical,  or  purely  rational,  the  NRC 
warns  in  "Behavioral  Modeling  &  Simulation."  The  NRC 
says  "understanding  emotion  is  critical  for  understanding 
cognition." 

Because  cognition  is  complex  and  because  emotions 
muddy  the  waters,  the  NRC  says  it's  common  to  observe 
seeming  paradoxes,  such  as  group  behavior  unlike 
individual  behavior.  A  classic  example  of  this  is  a  riot.  Riots, 
by  definition,  involve  a  lot  of  people,  but  most  participants 
wouldn't  loot  or  behave  violently  by  themselves. 

Generalizing  about  behavior  is  also  difficult,  the  NRC 
says  in  "Behavioral  Modeling  &  Simulation."  As  examples, 
the  NRC  notes  that  markets,  democracies,  and  traffic  laws 
work  differently  in  different  parts  of  the  world. 

Macro,  Meso,  Micro  Models 

The  analyst  has  to  be  prepared  to  attack  behavioral 
modeling  at  three  levels.  Macro  models  address 
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unemployment,  crime,  education,  poverty,  etc.  Micro 
models  address  individual  cognition,  e.g.,  psychological 
considerations  such  as  emotion  or  rational  choice.  But, 
as  the  NRC  says  in  "Behavioral  Modeling  &  Simulation," 
sometimes  the  answer  lies  in  the  interactions  between 
select  portions  of  the  macro  level  and  select  portions  of 
the  micro  level.  On  these  occasions,  the  analyst  needs  a 
"meso-level"  model. 

Agent-Based  Models 

The  value  of  agent-based  models  is  their  ability  to 
explore  "emergent  behavior"  -  behavior  that  wouldn't 
be  predictable  based  on  examination  of  the  entities  by 
themselves.  Sometimes  the  whole  is  different  from  the 
sum  of  its  parts,  and  agent-based  modeling  is  a  good 
way  to  look  at  how  interactions  between  agents  generate 
unexpected  results. 

The  key  with  agent-based  models  is  to  be  careful 
about  tradeoffs,  the  NRC  says  in  "Behavioral  Modeling  & 
Simulation."  The  sophistication  of  an  agent  in  an  agent- 
based  simulation  is  usually  inversely  proportional  to  the 
number  of  agents,  the  NRC  says.  The  analyst  is  asking  for 
trouble  if  he  tries  to  have  it  both  ways  -  a  model  with  lots 
of  agents  possessing  many  variables. 

It's  worth  noting  that  agent-based  models  work  best 
when  behavior  is  routine  or  benign.  For  example,  there 
are  a  number  of  useful  simulations  that  predict  pedestrian 
movement,  traffic  movement,  etc.  Such  simulations  can 
also  track  the  spread  of  a  disease  until  news  of  an  epidemic 
breaks  out.  Once  the  news  is  out,  however,  the  simulations 
aren't  very  useful.  As  a  speaker  at  the  World  Congress  on 
Social  Simulation  observed,  when  people  realize  there's  an 
epidemic,  you  have  a  panic  on  your  hands,  and  no  existing 
simulation  is  good  at  modeling  human  behavior  during  a 
panic. 

System  Dynamics  Models 

System  dynamics  models  emphasize  relationships  and 
lookfamiliartoengineers.They  have  blocks,  inputs,  outputs, 
and  feedback  loops.  System  dynamics  models  rest  upon 
differential  equations,  and  as  the  NRC  points  out,  there  can 
be  problems  when  modelers  work  with  social  scientists 
lacking  strong  mathematical  backgrounds.  Also,  as  the 
NRC  pointed  out  in  1998,  most  social  and  organizational 
theories  are  verbal  descriptions.  Marrying  a  complicated 
verbal  description  with  elaborate  mathematical  analysis  is 
destined  to  be  difficult. 


Network  Models 

Network  modeling  emphasizes  connections  from  one 
point  to  other  points,  i.e.,  from  node  to  node.  Social  network 
analysis,  which  is  closely  related  to  network  modeling,  has 
many  success  stories,  as  the  NRC  points  out  in  "Behavioral 
Modeling  &  Simulation."  These  successes  include  helping 
DOD  find  Saddam  Hussein,  helping  detectives  solve  crimes, 
and  helping  singles  with  Internet  dating. 

Network  models  and  the  analysis  they  support  look  at 
connections  to  better  understand  groups  and  individuals 
within  groups.  The  NRC  explains  a  number  of  pertinent 
terms  to  better  elucidate  network  analysis.  Some 
meaningful  terms  include: 

•  "Closeness  Centrality":  This  is  the  sum  of  the 

distances  from  one  node  to  all  other  nodes.  This  can 
help  explain  which  individuals  are  ensconced  in  a 
community  and  which  individuals  have  ties  beyond 
the  community. 

•  "Betweenness  Centrality":  This  describes  a  node 
that  is  on  many  of  a  network's  short  paths.  Typically, 
such  a  node  is  important  to  keeping  other  nodes 
informed,  which  means  disruption  of  this  node  can 
disrupt  the  whole  network. 

•  "Eigenvector  Centrality":  This  describes  the  extent 
to  which  a  node  is  connected  to  many  other  well- 
connected  notes. 

•  "Active  Area":  An  "Active  Area:  is  a  cohesive  subset  of 
nodes.  It  has  many  ties  within  the  cluster  and  few  ties 
outside  the  cluster. 

Network  analysis,  like  any  analysis  involving  behavioral 
models,  has  its  challenges.  For  example,  it's  difficult  to 
research  what  terrorists  are  doing,  which  means  it's  difficult 
to  know  whether  a  network  model  has  too  few/many  nodes 
or  connections.  What's  more,  the  NRC  notes,  it  isn't  known 
what  the  impact  is  when  an  analyst  studies  an  inaccurate 
model  of  a  terrorist  network. 

Another  problem  facing  network  modelers  is  "network 
overlap, "according  to  Dr.  Bill  Young,  a  Navy  analyst  who  has 
spent  the  bulk  of  his  career  as  an  anthropologist.  "Network 
overlap"  occurs  when  one  person  shows  up  in  two  or  three 
networks.  This  happens  frequently,  and  when  it  does,  it's 
often  necessary  to  know  how  the  person  prioritizes  his 
allegiances  to  the  various  networks.  Understanding  these 
allegiances  can  be  very  difficult  for  the  analyst  or  for  the 
researcher  gathering  data. 
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Capitalizing  on  the  strengths  of  network  analysis 
while  minimizing  the  weaknesses  is  a  topic  Jana  Diesner 
addressed  skillfully  at  the  recent  World  Congress  on  Social 
Simulation.  As  she  put  it,  the  analyst  can't  remove  himself 
from  the  analysis.  He  must  plunge  into  the  model  and  take 
ownership  of  the  conclusions.  Analysis  involving  social 
science  is  complex,  rests  upon  many  assumptions,  and 
includes  significant  uncertainty.  There  is  simply  no  way 
to  use  a  black  box  to  get  answers.  The  analyst  can't  hand 
the  problem  off  to  the  model;  instead,  the  analyst  must 
constantly  work  with  the  model  and  help  mold  the  product 
that  emerges  from  the  model.  Diesner  used  examples  of 
network  analysis  involving  millions  of  Enron  emails  to  show 
what  kinds  of  discoveries  can  be  made  and  the  importance 
of  an  analyst  willing  to  exercise  judgment  during  the  course 
of  a  study.  Diesner's  observations  deserve  consideration  on 
their  own  merits,  but  they  carry  extra  weight  because  she 
is  working  on  her  doctorate  at  Carnegie  Mellon  University 
under  the  supervision  of  Dr.  Kathleen  Carley,  who  helped 
author"Behavioral  Modeling  &  Simulation" for  the  NRC. 

V,  Validation,  &  A 

The  thorny  part  of  the  VV&A  process  for  behavioral 
models  is  validation.  How  do  you  establish  that  the  model 
tells  you  how  people  would  behave  in  the  real  world? 
As  the  NRC  says  in  "Behavioral  Modeling  &  Simulation," 
traditional  validation  -  something  akin  to  the  validation 
for  a  physics-based  model  -  is  "difficult  if  not  impossible  to 
fully  achieve."  However,  as  the  NRC  also  says,  this  isn't  an 
excuse  for  not  doing  anything.  Instead,  a  new  approach  to 
validation  is  in  order.  A  behavioral  model  should  be  judged 
on  how  well  it  narrows  the  range  of  possible  outcomes  for 
a  situation.  Validation  should  include  many  experts,  and  it 
should  be  conducted  in  light  of  similar  models  and  relevant 
studies.  A  behavioral  model  won't  be  predictive,  but  it  can 
be  informative. 

Conclusion 

We,  as  Marine  analysts,  must  be  skeptical  of  off-the-shelf 
solutions.  We  can't  treat  behavioral  models  as  black  boxes 
or  crystal  balls;  behavioral  models  simply  aren't  that  good. 
Instead,  analysts  must  incorporate  their  judgment  and 
their  understanding  into  whatever  answer  is  coming  out  of 
a  model. 

In  addition  to  lacking  off-the-shelf  models,  we  lack 
off-the-shelf  methods.  There  is  no  cookbook  recipe  for 
understanding  human  behavior.  Many  IW  problems  are 
unique  and  require  tailored  analysis,  observed  Scott  Moss 


during  the  recent  World  Congress  on  Social  Simulation.  As 
convenient  as  it  would  be  to  take  theory  off  the  shelf  and 
start  modeling,  you  can't  let  abstractions  obfuscate  the 
particulars  of  the  region  you  need  to  study,  said  Moss,  who 
was  educated  as  an  economist  but  has  spent  his  career  as 
a  sociologist.  "If  you  want  to  study  conflict  in  Baghdad," 
Moss  said,  "why  don't  you  start  by  studying  conflict  in 
Baghdad?" 

The  quest  for  an  all-purpose  tool  or  an  all-purpose 
method  has  always  charmed  DOD,  but  the  NRC  specifically 
warns  against  this  temptation  in  "Behavioral  Modeling 
&  Simulation."  Citing  the  overly  ambitious  master  plan 
devised  by  the  Defense  Modeling  &  Simulation  Office  in 
1995,  the  NRC  says  that  "developing  a  universal  ontological 
standard  for  model  creation  is  impractical." 

When  we  bring  modeling  expertise  to  bear  on  the 
analysis  of  IW,  we  must  remember  it  is  just  that  -  modeling 
expertise.  Behavioral  modeling  requires  what  its  name 
implies  it  requires  -  behavioral  expertise  and  modeling 
expertise.  What's  more,  behavioral  expertise  depends 
on  the  problem.  If  you're  trying  to  understand  market 
behavior,  an  economist  can  probably  help  you.  If  you're 
trying  to  understand  migration  in  the  face  of  genocide,  an 
economist  probably  can't  help  you.  Modelers  "operating 
outside  their  area  of  expertise"  have  a  tendency  to  create 
variables  "in  a  hodgepodge  fashion,"  the  NRC  says  in 
"Behavioral  Modeling  &  Simulation." 

It  also  seems  possible  that  modelers  operating  outside 
their  area  of  expertise  have  a  tendency  to  do  what  makes 
them  comfortable,  regardless  of  its  applicability.  This 
impression  arises  from  time  to  time  when  system  dynamics 
models  are  pitched  as  solutions  for  IW  analysis.  It  can  seem 
a  bit  odd  when  the  behavior  of  people  is  studied  with  the 
same  analytical  methods  applied  to  an  electrical  circuit 
full  of  capacitors  or  to  an  oscillating  weight  attached  to  a 
spring  and  a  damper. 

Along  the  same  lines,  there  sometimes  seems  to  be  a 
false  air  of  scientific  precision  when  models  are  touted  as 
being  "neural  network  models."The  definition  of  this  label 
sometimes  seems  to  exist  only  in  the  mind  of  the  beholder. 
Some  "neural  network  models"  claim  to  be  models  of  a 
human  nervous  system.  Others  claim  to  be  models  of 
something  else  that's  analogous  to  a  nervous  system. 
Either  way,  the  claims  should  be  met  with  careful  scrutiny. 
Physiologists  believe  the  human  nervous  system  may  have 
as  many  as  100  billion  neurons.  It  would  be  difficult  to 
take  that  complex  prototype  as  a  guideline  for  creating  a 
meaningful  model. 
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Modeling  Human  Behavior: 


The  fundamental  question  we  face  in  the  analysis  of  IW 
is  when  to  take  a  wide  view  and  when  to  take  a  deep  view. 
The  tradeoffs  are  obvious  -  a  wide  view  means  a  shallow 
view,  and  a  deep  view  means  a  narrow  view.  This  is  a 
difficult  question.  The  number  of  potential  trouble  spots 
on  the  globe  and  the  variety  of  social  sciences  seem  to 
be  a  vote  for  employing  a  wide  lens.  On  the  other  hand, 
the  complexities  of  human  behavior  seem  to  be  a  vote  for 
drilling  deeply  into  a  problem.  Ultimately,  it  seems  that 
drilling  deeply  is  too  risky.  That  means  putting  all  the  eggs 
in  one  basket.  History  shows  that  nations  are  very  bad 
at  predicting  what  the  next  war  will  look  like,  and  often 
nations  struggle  to  predict  where  the  next  war  will  be. 

To  fully  grasp  the  risks  of  drilling  deeply  into  a  single 
problem,  consider  what"Behavioral  Modeling &Simulation" 
said  about  agent-based  modeling.  As  the  NRC  put  it, 
"detailed,  sophisticated  ABM  frameworks  that  produce 
actionable  results  often  need  to  be  developed  by  a  team 
working  collectively  for  three  to  five  years."  Do  we  know 
where  we'll  be  fighting  in  three  to  five  years?  Do  we  know 
whether  the  problem  will  be  economic,  religious,  ethnic,  or 
something  else? 

We  need  to  trade  depth  for  agility  and  flexibility.  That 
doesn't  necessarily  mean  most  problems  will  be  analyzed 
with  models.  As  Dr.  Bill  Young,  a  Navy  analyst,  said,  the 
answer  to  a  quickly  arising  problem  might  be  something 
simple,  like  a  survey  of  SMEs  or  a  focus  group  of  SMEs.  That 
approach  is  far  quicker  than  modeling,  and  the  expertise  is 
more  firmly  rooted  in  the  specifics  of  the  problem. 

On  those  occasions  when  models  are  applied  to  IW,  the 
practical  move  is  to  use  simple  models  understood  by  the 
analysts  involved. 
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